perf(cli): improve the load time of eodag

[alambare]

lazy loading of imports an defering.

We gain x2 in init and basic usage of EODAG. And x10 on CLI helpers.

Before PR

$ time eodag 

real    0m3.507s
user    0m4.036s
sys     0m0.298s

$ time eodag list -p peps

real    0m9.771s
user    0m9.785s
sys     0m0.563s

Initialization of EODataAccessGateway():

Phase	duration
imports	2248.0ms

After PR

$ time eodag 

real    0m0.224s
user    0m0.174s
sys     0m0.047s

$ time eodag list -p peps

real    0m4.142s
user    0m4.628s
sys     0m0.323s

Initialization of EODataAccessGateway():

Phase	duration
imports	943.2ms

[github-actions]

Test Results

    4 files  ± 0      4 suites  ±0   3m 19s ⏱️ -1s
  697 tests + 8    696 ✅ + 8  1 💤 ±0  0 ❌ ±0 
2 830 runs  +32  2 824 ✅ +32  6 💤 ±0  0 ❌ ±0 

Results for commit 41c0144. ± Comparison against base commit d6875d8.

♻️ This comment has been updated with latest results.

[eodag-bot]

Code Coverage (Ubuntu)

Details

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     16       1  93.75%   63
cli.py                                         258      11  95.74%   99-110, 375, 657
config.py                                      305      24  92.13%   68-70, 73, 76, 79, 83, 87, 91-93, 581-583, 705-707, 726, 734, 766-771, 773
crunch.py                                        2       0  100.00%
api/__init__.py                                  0       0  100.00%
api/collection.py                              150       8  94.67%   176, 212, 215, 322, 361, 364, 382, 385
api/core.py                                    789      58  92.65%   265, 551, 599, 642, 682, 702, 743-748, 773, 857-876, 890, 896, 1049, 1054, 1153, 1192-1193, 1289-1290, 1316, 1347-1348, 1374, 1387, 1448-1449, 1466, 1480-1481, 1555-1560, 1572-1575, 1687, 2074, 2078, 2198, 2286-2287
api/provider.py                                381      35  90.81%   186, 190-191, 316-337, 405, 521, 525-526, 531-534, 545, 621-631, 821-822, 870, 877, 908-911, 947-948, 956-957
api/search_result.py                           181      19  89.50%   111, 123, 133, 154, 203, 220, 320, 375-378, 448, 453-454, 488, 502, 525-526, 532
api/product/__init__.py                         37       6  83.78%   50-53, 93, 95
api/product/_assets.py                          53       5  90.57%   97, 191-192, 195-199
api/product/_product.py                        254      20  92.13%   186, 307-308, 327-328, 439, 468, 475, 584, 604, 627-630, 639-642, 701, 769, 781
api/product/metadata_mapping.py                791      53  93.30%   125-127, 220-225, 249, 307-308, 396, 417, 469-470, 507, 528-531, 554, 566-567, 608, 631, 661-666, 731-736, 748, 756, 988, 1163, 1172-1176, 1193-1198, 1331, 1354, 1363, 1385, 1390, 1442, 1514, 1535, 1561, 1575, 1600, 1646, 1715, 1790
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     29       0  100.00%
api/product/drivers/generic.py                  11       0  100.00%
api/product/drivers/sentinel1.py                33       0  100.00%
api/product/drivers/sentinel2.py                33       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 25       7  72.00%   48, 55, 68-72
plugins/manager.py                             173      16  90.75%   102-107, 179, 201, 219-220, 232, 271-272, 372-375, 387-388
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                          101       8  92.08%   178-180, 228-229, 255-257
plugins/apis/usgs.py                           182      25  86.26%   157, 263, 297, 340-342, 347, 375-376, 381, 411-418, 429-434, 456-462
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 288-289, 339-343
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       6       0  100.00%
plugins/crunch/base.py                          12       0  100.00%
plugins/crunch/filter_date.py                   59       0  100.00%
plugins/crunch/filter_latest_intersect.py       54       6  88.89%   92-93, 100-101, 103-107
plugins/crunch/filter_latest_tpl_name.py        35       0  100.00%
plugins/crunch/filter_overlap.py                66      10  84.85%   118-121, 137-163
plugins/crunch/filter_property.py               30       0  100.00%
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        402      77  80.85%   270, 304, 353-356, 386-387, 395-399, 479-482, 522-524, 528, 559-560, 566-570, 601, 666-674, 738-832, 844-849, 887, 913, 958-960, 1012
plugins/download/base.py                       280      35  87.50%   133, 162, 309-310, 368-369, 411, 415-426, 440, 517-521, 551, 586-587, 612-621, 680, 701, 723, 731, 765
plugins/download/http.py                       564      77  86.35%   233, 275-278, 340-343, 346, 353-358, 389-391, 408, 423, 483, 518, 532, 546, 556-560, 576-581, 592, 611, 648-651, 672, 682, 689, 854, 886, 916-925, 961, 986-987, 1006-1011, 1020, 1035-1037, 1041, 1044, 1059-1060, 1144, 1196, 1238-1239, 1251, 1261, 1311-1312, 1342, 1362, 1403-1405, 1459-1460
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         196      18  90.82%   109, 113, 137-143, 200-203, 296, 317, 442, 492, 525-528, 537
plugins/search/build_search_result.py          506      84  83.40%   259-260, 296, 300, 320, 550-561, 576-578, 701, 725, 727, 794, 802-806, 827, 837, 863, 908, 932, 960, 978-993, 1043, 1068, 1071, 1075, 1084, 1090, 1128-1149, 1190, 1217-1218, 1227-1236, 1300, 1315, 1318, 1323, 1342-1351, 1472-1473, 1517, 1526-1528, 1585, 1633-1643
plugins/search/cop_marine.py                   268      56  79.10%   57, 65-67, 77-78, 83, 88-89, 105, 107, 110, 176-177, 220, 236, 242, 246, 250, 261, 272-273, 281, 313-316, 322, 343, 347, 351, 355, 359-363, 369-372, 375-392, 409-412, 465-469, 474, 486
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          112      87  22.32%   99-100, 104-105, 113-170, 176-189, 197-229, 246-287
plugins/search/qssearch.py                     824      95  88.47%   414-415, 532-533, 556-557, 569-573, 788-794, 852, 948, 955, 1026, 1047, 1050-1051, 1069, 1078-1079, 1106, 1177, 1186, 1191-1208, 1217, 1232, 1241-1244, 1254, 1276, 1365, 1388, 1461-1462, 1468, 1558, 1665-1669, 1735, 1738, 1742-1743, 1764-1767, 1779, 1801-1812, 1820, 1855-1857, 1880-1886, 1893, 1947, 1970, 1975-1976, 1991, 1997, 2007, 2097, 2101, 2112, 2136, 2149, 2157-2167, 2205-2209
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            84      18  78.57%   99-127, 166-169, 182, 224
resources/__init__.py                            0       0  100.00%
resources/shp/__init__.py                        0       0  100.00%
types/__init__.py                              167      43  74.25%   58, 62, 71-75, 86-98, 126-128, 135-140, 216, 219, 257, 267-283, 288, 290, 312, 317, 325, 335
types/bbox.py                                   39      19  51.28%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                          10       0  100.00%
types/queryables.py                            112       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/stac_extensions.py                       114       1  99.12%   282
types/stac_metadata.py                         118      16  86.44%   94, 115-116, 149, 179-193, 202-209
utils/__init__.py                              600      32  94.67%   69, 215, 248, 328-332, 407-411, 491, 531-532, 561, 943-946, 997, 1016-1017, 1049, 1067-1068, 1187, 1277, 1447, 1536, 1551, 1560, 1572, 1762
utils/cache.py                                  22       0  100.00%
utils/dates.py                                 113       4  96.46%   171, 329-331
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       0  100.00%
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      12  95.00%   200-203, 245, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                        10142    1238  87.79%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
__init__.py                                 +8      +1  -6.25%
cli.py                                      +7       0  +0.12%
api/core.py                                 +2      +1  -0.11%
api/product/__init__.py                    +19      +4  -5.11%
api/product/_assets.py                      +1      +1  -1.74%
api/product/_product.py                     +6       0  +0.19%
plugins/download/aws.py                     +1      +1  -0.20%
plugins/search/build_search_result.py       +2      +1  -0.13%
plugins/search/qssearch.py                  +1      +1  -0.11%
utils/__init__.py                           -8     -11  +1.74%
TOTAL                                      +39      -1  +0.05%

Results for commit: 41c0144

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

[eodag-bot]

Code Coverage (Windows)

Details

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     16       1  93.75%   63
cli.py                                         258      11  95.74%   99-110, 375, 657
config.py                                      305      24  92.13%   68-70, 73, 76, 79, 83, 87, 91-93, 581-583, 705-707, 726, 734, 766-771, 773
crunch.py                                        2       0  100.00%
api/__init__.py                                  0       0  100.00%
api/collection.py                              150       8  94.67%   176, 212, 215, 322, 361, 364, 382, 385
api/core.py                                    789      58  92.65%   265, 551, 599, 642, 682, 702, 743-748, 773, 857-876, 890, 896, 1049, 1054, 1153, 1192-1193, 1289-1290, 1316, 1347-1348, 1374, 1387, 1448-1449, 1466, 1480-1481, 1555-1560, 1572-1575, 1687, 2074, 2078, 2198, 2286-2287
api/provider.py                                381      35  90.81%   186, 190-191, 316-337, 405, 521, 525-526, 531-534, 545, 621-631, 821-822, 870, 877, 908-911, 947-948, 956-957
api/search_result.py                           181      19  89.50%   111, 123, 133, 154, 203, 220, 320, 375-378, 448, 453-454, 488, 502, 525-526, 532
api/product/__init__.py                         37       6  83.78%   50-53, 93, 95
api/product/_assets.py                          53       5  90.57%   97, 191-192, 195-199
api/product/_product.py                        254      20  92.13%   186, 307-308, 327-328, 439, 468, 475, 584, 604, 627-630, 639-642, 701, 769, 781
api/product/metadata_mapping.py                791      53  93.30%   125-127, 220-225, 249, 307-308, 396, 417, 469-470, 507, 528-531, 554, 566-567, 608, 631, 661-666, 731-736, 748, 756, 988, 1163, 1172-1176, 1193-1198, 1331, 1354, 1363, 1385, 1390, 1442, 1514, 1535, 1561, 1575, 1600, 1646, 1715, 1790
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     29       0  100.00%
api/product/drivers/generic.py                  11       0  100.00%
api/product/drivers/sentinel1.py                33       0  100.00%
api/product/drivers/sentinel2.py                33       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 25       7  72.00%   48, 55, 68-72
plugins/manager.py                             173      16  90.75%   102-107, 179, 201, 219-220, 232, 271-272, 372-375, 387-388
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                          101       8  92.08%   178-180, 228-229, 255-257
plugins/apis/usgs.py                           182      25  86.26%   157, 263, 297, 340-342, 347, 375-376, 381, 411-418, 429-434, 456-462
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 288-289, 339-343
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       6       0  100.00%
plugins/crunch/base.py                          12       0  100.00%
plugins/crunch/filter_date.py                   59       0  100.00%
plugins/crunch/filter_latest_intersect.py       54       6  88.89%   92-93, 100-101, 103-107
plugins/crunch/filter_latest_tpl_name.py        35       0  100.00%
plugins/crunch/filter_overlap.py                66      10  84.85%   118-121, 137-163
plugins/crunch/filter_property.py               30       0  100.00%
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        402      77  80.85%   270, 304, 353-356, 386-387, 395-399, 479-482, 522-524, 528, 559-560, 566-570, 601, 666-674, 738-832, 844-849, 887, 913, 958-960, 1012
plugins/download/base.py                       280      37  86.79%   133, 162, 229-231, 309-310, 368-369, 411, 415-426, 440, 517-521, 551, 586-587, 612-621, 680, 701, 723, 731, 765
plugins/download/http.py                       564      77  86.35%   233, 275-278, 340-343, 346, 353-358, 389-391, 408, 423, 483, 518, 532, 546, 556-560, 576-581, 592, 611, 648-651, 672, 682, 689, 854, 886, 916-925, 961, 986-987, 1006-1011, 1020, 1035-1037, 1041, 1044, 1059-1060, 1144, 1196, 1238-1239, 1251, 1261, 1311-1312, 1342, 1362, 1403-1405, 1459-1460
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         196      18  90.82%   109, 113, 137-143, 200-203, 296, 317, 442, 492, 525-528, 537
plugins/search/build_search_result.py          506      84  83.40%   259-260, 296, 300, 320, 550-561, 576-578, 701, 725, 727, 794, 802-806, 827, 837, 863, 908, 932, 960, 978-993, 1043, 1068, 1071, 1075, 1084, 1090, 1128-1149, 1190, 1217-1218, 1227-1236, 1300, 1315, 1318, 1323, 1342-1351, 1472-1473, 1517, 1526-1528, 1585, 1633-1643
plugins/search/cop_marine.py                   268      56  79.10%   57, 65-67, 77-78, 83, 88-89, 105, 107, 110, 176-177, 220, 236, 242, 246, 250, 261, 272-273, 281, 313-316, 322, 343, 347, 351, 355, 359-363, 369-372, 375-392, 409-412, 465-469, 474, 486
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          112      87  22.32%   99-100, 104-105, 113-170, 176-189, 197-229, 246-287
plugins/search/qssearch.py                     824      95  88.47%   414-415, 532-533, 556-557, 569-573, 788-794, 852, 948, 955, 1026, 1047, 1050-1051, 1069, 1078-1079, 1106, 1177, 1186, 1191-1208, 1217, 1232, 1241-1244, 1254, 1276, 1365, 1388, 1461-1462, 1468, 1558, 1665-1669, 1735, 1738, 1742-1743, 1764-1767, 1779, 1801-1812, 1820, 1855-1857, 1880-1886, 1893, 1947, 1970, 1975-1976, 1991, 1997, 2007, 2097, 2101, 2112, 2136, 2149, 2157-2167, 2205-2209
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            84      18  78.57%   99-127, 166-169, 182, 224
resources/__init__.py                            0       0  100.00%
resources/shp/__init__.py                        0       0  100.00%
types/__init__.py                              167      43  74.25%   58, 62, 71-75, 86-98, 126-128, 135-140, 216, 219, 257, 267-283, 288, 290, 312, 317, 325, 335
types/bbox.py                                   39      19  51.28%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                          10       0  100.00%
types/queryables.py                            112       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/stac_extensions.py                       114       1  99.12%   282
types/stac_metadata.py                         118      16  86.44%   94, 115-116, 149, 179-193, 202-209
utils/__init__.py                              600      32  94.67%   69, 215, 248, 328-332, 407-411, 491, 531-532, 561, 943-946, 997, 1016-1017, 1049, 1067-1068, 1187, 1277, 1447, 1536, 1551, 1560, 1572, 1762
utils/cache.py                                  22       0  100.00%
utils/dates.py                                 113       4  96.46%   171, 329-331
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       0  100.00%
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      12  95.00%   200-203, 245, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                        10142    1240  87.77%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
__init__.py                                 +8      +1  -6.25%
cli.py                                      +7       0  +0.12%
api/core.py                                 +2      +1  -0.11%
api/product/__init__.py                    +19      +4  -5.11%
api/product/_assets.py                      +1      +1  -1.74%
api/product/_product.py                     +6       0  +0.19%
plugins/download/aws.py                     +1      +1  -0.20%
plugins/search/build_search_result.py       +2      +1  -0.13%
plugins/search/qssearch.py                  +1      +1  -0.11%
utils/__init__.py                           -8     -11  +1.74%
TOTAL                                      +39      -1  +0.05%

Results for commit: 41c0144

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results