[GoogleSheet] Add a new Google Sheet loader #1609
+5,862
−65
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| GOOGLE_SPREADSHEET_NAME='Flow test spreadsheet' | ||
| GOOGLE_SHEET_NAME='Flow test sheet' | ||
| GOOGLE_SHEET_EMAIL='random@gmail.com' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| .env | ||
| auth.json | ||
| vendor |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "type": "service_account", | ||
| "project_id": "flow-test", | ||
| "client_email": "flow-test@flow-test.iam.gserviceaccount.com", | ||
| "client_id": "1234567890", | ||
| "auth_uri": "https://accounts.google.com/o/oauth2/auth", | ||
| "token_uri": "https://oauth2.googleapis.com/token", | ||
| "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", | ||
| "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/flow-test%flow-test.iam.gserviceaccount.com", | ||
| "universe_domain": "googleapis.com" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| use function Flow\ETL\Adapter\GoogleSheet\{from_google_sheet, google_create_spreadsheet, google_sheets, to_google_sheet}; | ||
| use function Flow\ETL\DSL\{data_frame, from_array, to_stream}; | ||
| use Google\Service\Sheets; | ||
| use Symfony\Component\Dotenv\Dotenv; | ||
|
|
||
| require __DIR__ . '/vendor/autoload.php'; | ||
|
|
||
| if (!\file_exists(__DIR__ . '/auth.json')) { | ||
| print 'Example skipped. Please create .env file with Google Auth credentials.' . PHP_EOL; | ||
|
|
||
| return; | ||
| } | ||
|
|
||
| if (!\file_exists(__DIR__ . '/.env')) { | ||
| print 'Example skipped. Please create .env file with Google Sheet details.' . PHP_EOL; | ||
|
|
||
| return; | ||
| } | ||
|
|
||
| $dotenv = new Dotenv(); | ||
| $dotenv->load(__DIR__ . '/.env'); | ||
|
|
||
| $service = google_sheets( | ||
| \json_decode((string) \file_get_contents(__DIR__ . '/auth.json'), true, 512, JSON_THROW_ON_ERROR), | ||
| Sheets::SPREADSHEETS | ||
| ); | ||
|
|
||
| $spreadsheet = google_create_spreadsheet( | ||
| $service, | ||
| $_ENV['GOOGLE_SPREADSHEET_NAME'], | ||
| $sheetName = $_ENV['GOOGLE_SHEET_EMAIL'], | ||
| $_ENV['GOOGLE_SHEET_EMAIL'] | ||
| ); | ||
|
|
||
| data_frame() | ||
| ->read(from_array([ | ||
| ['id' => 1, 'text' => 'lorem ipsum'], | ||
| ['id' => 2, 'text' => 'lorem ipsum'], | ||
| ['id' => 3, 'text' => 'lorem ipsum'], | ||
| ['id' => 4, 'text' => 'lorem ipsum'], | ||
| ['id' => 5, 'text' => 'lorem ipsum'], | ||
| ['id' => 6, 'text' => 'lorem ipsum'], | ||
| ])) | ||
| ->write(to_google_sheet($service, $spreadsheet->spreadsheetId, $sheetName)) | ||
| ->run(); | ||
|
|
||
| data_frame() | ||
| ->read(from_google_sheet($service, $spreadsheet->spreadsheetId, $sheetName)) | ||
| ->write(to_stream(__DIR__ . '/output.txt', truncate: false)) | ||
| ->run(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "name": "flow-php/examples", | ||
| "description": "Flow PHP - Examples", | ||
| "license": "MIT", | ||
| "type": "library", | ||
| "require": { | ||
| "flow-php/etl": "1.x-dev", | ||
| "flow-php/etl-adapter-google-sheet": "1.x-dev", | ||
| "symfony/dotenv": "^7.2" | ||
| }, | ||
| "config": { | ||
| "allow-plugins": { | ||
| "php-http/discovery": false | ||
| } | ||
| }, | ||
| "archive": { | ||
| "exclude": [ | ||
| ".env", | ||
| "auth.json.dist", | ||
| "vendor" | ||
| ] | ||
| } | ||
| } |
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 1 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 2 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 3 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 4 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 5 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 6 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| GOOGLE_SPREADSHEET_NAME='Flow test spreadsheet' | ||
| GOOGLE_SHEET_NAME='Flow test sheet' | ||
| GOOGLE_SHEET_EMAIL='random@gmail.com' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| .env | ||
| auth.json | ||
| vendor |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "type": "service_account", | ||
| "project_id": "flow-test", | ||
| "client_email": "flow-test@flow-test.iam.gserviceaccount.com", | ||
| "client_id": "1234567890", | ||
| "auth_uri": "https://accounts.google.com/o/oauth2/auth", | ||
| "token_uri": "https://oauth2.googleapis.com/token", | ||
| "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", | ||
| "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/flow-test%flow-test.iam.gserviceaccount.com", | ||
| "universe_domain": "googleapis.com" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| use function Flow\ETL\Adapter\GoogleSheet\{from_google_sheet, google_create_spreadsheet, google_sheets, to_google_sheet}; | ||
| use function Flow\ETL\DSL\{data_frame, from_array, to_stream}; | ||
| use Google\Service\Sheets; | ||
| use Symfony\Component\Dotenv\Dotenv; | ||
|
|
||
| require __DIR__ . '/vendor/autoload.php'; | ||
|
|
||
| if (!\file_exists(__DIR__ . '/auth.json')) { | ||
| print 'Example skipped. Please create .env file with Google Auth credentials.' . PHP_EOL; | ||
|
|
||
| return; | ||
| } | ||
|
|
||
| if (!\file_exists(__DIR__ . '/.env')) { | ||
| print 'Example skipped. Please create .env file with Google Sheet details.' . PHP_EOL; | ||
|
|
||
| return; | ||
| } | ||
|
|
||
| $dotenv = new Dotenv(); | ||
| $dotenv->load(__DIR__ . '/.env'); | ||
|
|
||
| $service = google_sheets( | ||
| \json_decode((string) \file_get_contents(__DIR__ . '/auth.json'), true, 512, JSON_THROW_ON_ERROR), | ||
| Sheets::SPREADSHEETS | ||
| ); | ||
|
|
||
| $spreadsheet = google_create_spreadsheet( | ||
| $service, | ||
| $_ENV['GOOGLE_SPREADSHEET_NAME'], | ||
| $sheetName = $_ENV['GOOGLE_SHEET_EMAIL'], | ||
| $_ENV['GOOGLE_SHEET_EMAIL'] | ||
| ); | ||
|
|
||
| data_frame() | ||
| ->read(from_array([ | ||
| ['id' => 1, 'text' => 'lorem ipsum'], | ||
| ['id' => 2, 'text' => 'lorem ipsum'], | ||
| ['id' => 3, 'text' => 'lorem ipsum'], | ||
| ['id' => 4, 'text' => 'lorem ipsum'], | ||
| ['id' => 5, 'text' => 'lorem ipsum'], | ||
| ['id' => 6, 'text' => 'lorem ipsum'], | ||
| ])) | ||
| ->write(to_google_sheet($service, $spreadsheet->spreadsheetId, $sheetName)) | ||
| ->run(); | ||
|
|
||
| data_frame() | ||
| ->read(from_google_sheet($service, $spreadsheet->spreadsheetId, $sheetName)) | ||
| ->write(to_stream(__DIR__ . '/output.txt', truncate: false)) | ||
| ->run(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "name": "flow-php/examples", | ||
| "description": "Flow PHP - Examples", | ||
| "license": "MIT", | ||
| "type": "library", | ||
| "require": { | ||
| "flow-php/etl": "1.x-dev", | ||
| "flow-php/etl-adapter-google-sheet": "1.x-dev", | ||
| "symfony/dotenv": "^7.2" | ||
| }, | ||
| "config": { | ||
| "allow-plugins": { | ||
| "php-http/discovery": false | ||
| } | ||
| }, | ||
| "archive": { | ||
| "exclude": [ | ||
| ".env", | ||
| "auth.json.dist", | ||
| "vendor" | ||
| ] | ||
| } | ||
| } |
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 1 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 2 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 3 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 4 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 5 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows | ||
| +----+-------------+ | ||
| | id | text | | ||
| +----+-------------+ | ||
| | 6 | lorem ipsum | | ||
| +----+-------------+ | ||
| 1 rows |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,4 @@ | ||
| vendor | ||
| *.cache | ||
| var | ||
| auth.json.dist |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet; | ||
|
|
||
| use Flow\ETL\{Adapter\GoogleSheet\RowsNormalizer\EntryNormalizer, | ||
| Adapter\GoogleSheet\Spreadsheet\SpreadsheetManager, | ||
| FlowContext, | ||
| Loader, | ||
| Row\Entry, | ||
| Rows}; | ||
| use Flow\ETL\Loader\Closure; | ||
| use Google\Service\Sheets; | ||
| use Google\Service\Sheets\{ValueRange}; | ||
|
|
||
| final class GoogleSheetLoader implements Closure, Loader | ||
| { | ||
| private string $dateFormat = 'Y-m-d'; | ||
|
|
||
| private string $dateTimeFormat = 'Y-m-d H:i:s'; | ||
|
|
||
| private ValueInputOption $inputOption = ValueInputOption::USER_ENTERED; | ||
|
|
||
| private int $loadedRows = 0; | ||
|
|
||
| private int $sheetDynamicAllocationSize = 1_000; | ||
|
|
||
| private bool $withHeader = true; | ||
|
|
||
| public function __construct( | ||
| private readonly Sheets $service, | ||
| private readonly string $spreadsheetId, | ||
| private readonly string $sheetName, | ||
stloyd marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ) { | ||
| } | ||
|
|
||
| public function closure(FlowContext $context) : void | ||
| { | ||
| $this->loadedRows = 0; | ||
| } | ||
|
|
||
| public function load(Rows $rows, FlowContext $context) : void | ||
| { | ||
| if (!$rows->count()) { | ||
| return; | ||
| } | ||
|
|
||
| $manager = new SpreadsheetManager($this->service, $this->spreadsheetId); | ||
| $spreadsheetProperties = $manager->getSpreadsheetProperties($this->sheetName); | ||
|
|
||
| $columnsCount = $rows->first()->entries()->count(); | ||
| $currentRow = $this->calculateCurrentRow($rows); | ||
| $sheetRange = $this->calculateSheetAddress($columnsCount, $currentRow); | ||
|
|
||
| // Calculate if spreadsheet size fits current load | ||
| if ($rows->count() >= $spreadsheetProperties->rowCount) { | ||
| $manager->increaseSheetSize($currentRow, $columnsCount, $spreadsheetProperties->sheetId, $this->sheetDynamicAllocationSize); | ||
| } | ||
|
|
||
| $normalizer = new RowsNormalizer( | ||
| new EntryNormalizer($this->dateTimeFormat, $this->dateFormat) | ||
| ); | ||
|
|
||
| $this->service->spreadsheets_values->update( | ||
| $this->spreadsheetId, | ||
| $sheetRange->toString(), | ||
| new ValueRange( | ||
| [ | ||
| 'values' => $this->values($rows, $normalizer), | ||
| 'majorDimension' => 'ROWS', | ||
| ] | ||
| ), | ||
| $this->inputOption->toArray() | ||
| ); | ||
|
|
||
| $this->loadedRows = $currentRow; | ||
| } | ||
|
|
||
| public function withDateFormat(string $dateFormat) : self | ||
| { | ||
| $this->dateFormat = $dateFormat; | ||
|
|
||
| return $this; | ||
| } | ||
|
|
||
| public function withDateTimeFormat(string $dateTimeFormat) : self | ||
| { | ||
| $this->dateTimeFormat = $dateTimeFormat; | ||
|
|
||
| return $this; | ||
| } | ||
|
|
||
| public function withHeader(bool $withHeader) : self | ||
| { | ||
| $this->withHeader = $withHeader; | ||
|
|
||
| return $this; | ||
| } | ||
|
|
||
| public function withInputOption(ValueInputOption $inputOption) : self | ||
| { | ||
| $this->inputOption = $inputOption; | ||
|
|
||
| return $this; | ||
| } | ||
|
|
||
| public function withSheetDynamicAllocationSize(int $bySize) : self | ||
| { | ||
| $this->sheetDynamicAllocationSize = $bySize; | ||
|
|
||
| return $this; | ||
| } | ||
|
|
||
| private function calculateCurrentRow(Rows $rows) : int | ||
| { | ||
| return $rows->count() + (0 === $this->loadedRows ? ($this->withHeader ? 1 : 0) : $this->loadedRows); | ||
| } | ||
|
|
||
| private function calculateSheetAddress(int $columnsCount, int $currentRow) : SheetAddress | ||
| { | ||
| return SheetAddress::calculate( | ||
| $columnsCount, | ||
| $currentRow, | ||
| new SheetAddress( | ||
| startCell: 0 === $this->loadedRows ? 'A1' : 'A' . $this->loadedRows + 1, | ||
| sheetName: $this->sheetName | ||
| ) | ||
| ); | ||
| } | ||
|
|
||
| private function values(Rows $rows, RowsNormalizer $normalizer) : array | ||
| { | ||
| $values = []; | ||
|
|
||
| if ($this->withHeader && 0 === $this->loadedRows) { | ||
| $values[] = $rows->first()->entries()->map(fn (Entry $entry) => $entry->name()); | ||
| } | ||
|
|
||
| foreach ($normalizer->normalize($rows) as $normalizedRow) { | ||
| $values[] = $normalizedRow; | ||
| } | ||
|
|
||
| return $values; | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet; | ||
|
|
||
| use Flow\ETL\Adapter\GoogleSheet\RowsNormalizer\EntryNormalizer; | ||
| use Flow\ETL\Rows; | ||
|
|
||
| final readonly class RowsNormalizer | ||
| { | ||
| public function __construct(private EntryNormalizer $entryNormalizer) | ||
| { | ||
| } | ||
|
|
||
| /** | ||
| * @return \Generator<array<null|bool|float|int|string>> | ||
| */ | ||
| public function normalize(Rows $rows) : \Generator | ||
| { | ||
| foreach ($rows as $row) { | ||
| $normalizedRow = []; | ||
|
|
||
| foreach ($row->entries() as $entry) { | ||
| $normalizedRow[] = $this->entryNormalizer->normalize($entry); | ||
| } | ||
|
|
||
| yield $normalizedRow; | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\RowsNormalizer; | ||
|
|
||
| use function Flow\Types\DSL\type_json; | ||
| use Flow\ETL\Row\Entry; | ||
| use Flow\ETL\Row\Entry\{DateEntry, DateTimeEntry, EnumEntry, JsonEntry, ListEntry, MapEntry, StructureEntry, TimeEntry, UuidEntry, XMLElementEntry, XMLEntry}; | ||
|
|
||
| final readonly class EntryNormalizer | ||
| { | ||
| public function __construct( | ||
| private string $dateTimeFormat, | ||
| private string $dateFormat, | ||
| ) { | ||
| } | ||
|
|
||
| /** | ||
| * @param Entry<mixed, mixed> $entry | ||
| */ | ||
| public function normalize(Entry $entry) : string|float|int|bool|null | ||
| { | ||
| return match ($entry::class) { | ||
| UuidEntry::class, | ||
| XMLElementEntry::class, | ||
| XMLEntry::class => $entry->toString(), | ||
| DateTimeEntry::class => $entry->value() === null ? '' : $entry->value()->format($this->dateTimeFormat), | ||
| DateEntry::class => $entry->value() === null ? '' : $entry->value()->format($this->dateFormat), | ||
| TimeEntry::class => $entry->toString(), | ||
| EnumEntry::class => $entry->value()?->name, | ||
| ListEntry::class, | ||
| MapEntry::class, | ||
| StructureEntry::class, | ||
| JsonEntry::class => type_json()->cast($entry->value()), | ||
| default => $entry->value() ?? '', | ||
| }; | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet; | ||
|
|
||
| final readonly class SheetAddress | ||
| { | ||
| private ?string $endCell; | ||
|
|
||
| private string $startCell; | ||
|
|
||
| public function __construct( | ||
| string $startCell = 'A1', | ||
| ?string $endCell = null, | ||
| private ?string $sheetName = null, | ||
| ) { | ||
| $this->startCell = \strtoupper($startCell); | ||
| $this->endCell = $endCell ? \strtoupper($endCell) : null; | ||
| } | ||
|
|
||
| public static function calculate(int $columns, int $rows, ?self $startAddress = null) : self | ||
| { | ||
| $startAddress ??= new self('A1'); | ||
| $startColumn = $startAddress->getStartCellColumn(); | ||
| $startRow = $startAddress->getStartCellRow(); | ||
|
|
||
| $startColumnNumber = 0; | ||
|
|
||
| for ($i = 0, $length = \strlen($startColumn); $i < $length; $i++) { | ||
| $startColumnNumber = $startColumnNumber * 26 + (ord($startColumn[$i]) - 64); | ||
| } | ||
|
|
||
| $endColumnNumber = $startColumnNumber + $columns - 1; | ||
| $endColumn = ''; | ||
|
|
||
| while ($endColumnNumber > 0) { | ||
| $remainder = ($endColumnNumber - 1) % 26; | ||
| $endColumn = chr(65 + $remainder) . $endColumn; | ||
| $endColumnNumber = floor(($endColumnNumber - $remainder) / 26); | ||
| } | ||
|
|
||
| return new self( | ||
| $startColumn . $startRow, | ||
| $endColumn . ($startRow + $rows - 1), | ||
| $startAddress->getSheetName() | ||
| ); | ||
| } | ||
|
|
||
| public function getEndCell() : ?string | ||
| { | ||
| return $this->endCell; | ||
| } | ||
|
|
||
| public function getEndCellColumn() : string | ||
| { | ||
| return (string) preg_replace('/[0-9]/', '', (string) $this->endCell); | ||
| } | ||
|
|
||
| public function getEndCellRow() : int | ||
| { | ||
| return (int) preg_replace('/[A-Z]/', '', (string) $this->endCell); | ||
| } | ||
|
|
||
| public function getSheetName() : ?string | ||
| { | ||
| return $this->sheetName; | ||
| } | ||
|
|
||
| public function getStartCell() : string | ||
| { | ||
| return $this->startCell; | ||
| } | ||
|
|
||
| public function getStartCellColumn() : string | ||
| { | ||
| return (string) preg_replace('/[0-9]/', '', $this->startCell); | ||
| } | ||
|
|
||
| public function getStartCellRow() : int | ||
| { | ||
| return (int) preg_replace('/[A-Z]/', '', $this->startCell); | ||
| } | ||
|
|
||
| public function toString() : string | ||
| { | ||
| return ($this->sheetName ? ("'" . $this->sheetName . "'!") : '') . $this->startCell . ($this->endCell ? ':' . $this->endCell : ''); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Spreadsheet; | ||
|
|
||
| use Flow\ETL\Exception\InvalidArgumentException; | ||
| use Google\Service\Sheets; | ||
| use Google\Service\Sheets\{BatchUpdateSpreadsheetRequest, Spreadsheet}; | ||
|
|
||
| final class SpreadsheetManager | ||
| { | ||
| private ?Spreadsheet $spreadsheet = null; | ||
|
|
||
| public function __construct( | ||
| private readonly Sheets $service, | ||
| private readonly string $spreadsheetId, | ||
| ) { | ||
| } | ||
|
|
||
| public function getSpreadsheetProperties(string $sheetName) : SpreadsheetProperties | ||
| { | ||
| $rowCount = 0; | ||
| $sheetId = null; | ||
|
|
||
| foreach ($this->spreadsheet()->getSheets() as $sheet) { | ||
| if ($sheetName === $sheet->getProperties()->getTitle()) { | ||
| $sheetId = $sheet->getProperties()->getSheetId(); | ||
| $gridProperties = $sheet->getProperties()->getGridProperties(); | ||
| $rowCount += $gridProperties->getRowCount(); | ||
|
|
||
| break; | ||
| } | ||
| } | ||
|
|
||
| if (null === $sheetId) { | ||
| throw new InvalidArgumentException("Sheet '{$sheetName}' not found in spreadsheet '{$this->spreadsheetId}'"); | ||
| } | ||
|
|
||
| return new SpreadsheetProperties($sheetId, $rowCount); | ||
| } | ||
|
|
||
| public function increaseSheetSize(int $currentRow, int $columnsCount, int $sheetId, int $sheetDynamicAllocationSize) : void | ||
|
Check warning on line 43 in src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/Spreadsheet/SpreadsheetManager.php
|
||
| { | ||
| $this->service->spreadsheets->batchUpdate( | ||
| $this->spreadsheet()->spreadsheetId, | ||
| new BatchUpdateSpreadsheetRequest( | ||
| [ | ||
| 'requests' => [ | ||
| [ | ||
| 'updateSheetProperties' => [ | ||
| 'properties' => [ | ||
| 'sheetId' => $sheetId, | ||
| 'gridProperties' => [ | ||
| 'rowCount' => $currentRow + $sheetDynamicAllocationSize, | ||
|
Check warning on line 55 in src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/Spreadsheet/SpreadsheetManager.php
|
||
| // Ensure we update the column size as well | ||
| 'columnCount' => $columnsCount, | ||
| ], | ||
| ], | ||
| 'fields' => 'gridProperties(rowCount,columnCount)', | ||
| ], | ||
| ], | ||
| ], | ||
| ] | ||
| ) | ||
| ); | ||
|
Check warning on line 66 in src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/Spreadsheet/SpreadsheetManager.php
|
||
| } | ||
|
|
||
| private function spreadsheet() : Spreadsheet | ||
| { | ||
| if ($this->spreadsheet !== null) { | ||
| return $this->spreadsheet; | ||
|
Check warning on line 72 in src/adapter/etl-adapter-google-sheet/src/Flow/ETL/Adapter/GoogleSheet/Spreadsheet/SpreadsheetManager.php
|
||
| } | ||
|
|
||
| return $this->spreadsheet = $this->service->spreadsheets->get($this->spreadsheetId); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Spreadsheet; | ||
|
|
||
| final class SpreadsheetProperties | ||
| { | ||
| public function __construct( | ||
| public int $sheetId, | ||
| public int $rowCount, | ||
| ) { | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet; | ||
|
|
||
| enum ValueInputOption : string | ||
| { | ||
| case RAW = 'RAW'; | ||
| case USER_ENTERED = 'USER_ENTERED'; | ||
|
|
||
| public function toArray() : array | ||
| { | ||
| return ['valueInputOption' => $this->value]; | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "spreadsheetId": "1234567890", | ||
| "properties": { | ||
| "title": "Test Spreadsheet", | ||
| "locale": "en_US", | ||
| "timeZone": "America/Los_Angeles" | ||
| }, | ||
| "sheets": [ | ||
| { | ||
| "properties": { | ||
| "sheetId": 0, | ||
| "title": "Sheet", | ||
| "index": 0, | ||
| "sheetType": "GRID", | ||
| "gridProperties": { | ||
| "rowCount": 1000, | ||
| "columnCount": 26 | ||
| } | ||
| } | ||
| } | ||
| ], | ||
| "spreadsheetUrl": "https://docs.google.com/spreadsheets/d/1234567890/edit" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "spreadsheetId": "1234567890", | ||
| "updatedRange": "Sheet!A1:D4", | ||
| "updatedRows": 4, | ||
| "updatedColumns": 2, | ||
| "updatedCells": 8, | ||
| "updatedData": { | ||
| "range": "Sheet!A1:C4", | ||
| "majorDimension": "ROWS", | ||
| "values": [ | ||
| ["id", "name"], | ||
| [12345, "Norbert"], | ||
| [54321, "Joseph"], | ||
| [666, "Dominik"] | ||
| ] | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests; | ||
|
|
||
| use GuzzleHttp\{Client as HttpClient, Handler\MockHandler, Psr7\Request}; | ||
| use GuzzleHttp\Psr7\Response; | ||
| use PHPUnit\Framework\Assert; | ||
|
|
||
| final class HttpClientContext | ||
| { | ||
| public function __construct(private array $queue = []) | ||
| { | ||
| } | ||
|
|
||
| public function add( | ||
| HttpRequestContext $requestContext, | ||
| string $fixtureFile, | ||
| ) : self { | ||
| $this->queue[] = function (Request $request) use ($requestContext, $fixtureFile) : Response { | ||
| Assert::assertSame($requestContext->method, $request->getMethod()); | ||
| Assert::assertSame($requestContext->url, (string) $request->getUri()); | ||
|
|
||
| if (null !== $requestContext->body) { | ||
| Assert::assertSame($requestContext->body, (string) $request->getBody()); | ||
| } | ||
|
|
||
| return new Response( | ||
| headers: ['Content-Type' => 'application/json'], | ||
| body: file_get_contents($fixtureFile) ?: throw new \RuntimeException('Failed to read file: ' . $fixtureFile) | ||
| ); | ||
| }; | ||
|
|
||
| return $this; | ||
| } | ||
|
|
||
| public function createHttpClient() : HttpClient | ||
| { | ||
| return new HttpClient( | ||
| [ | ||
| 'handler' => MockHandler::createWithMiddleware($this->queue), | ||
| ] | ||
| ); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests; | ||
|
|
||
| final class HttpRequestContext | ||
| { | ||
| public function __construct( | ||
| public string $method, | ||
| public string $url, | ||
| public ?string $body = null, | ||
| ) { | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests\Integration; | ||
|
|
||
| use function Flow\ETL\Adapter\GoogleSheet\to_google_sheet; | ||
| use function Flow\ETL\DSL\{config, flow_context, int_entry, row, rows, string_entry}; | ||
| use Flow\ETL\Adapter\GoogleSheet\{ | ||
| Tests\GoogleSheetsContext, | ||
| Tests\HttpClientContext, | ||
| Tests\HttpRequestContext}; | ||
| use Flow\ETL\Tests\FlowTestCase; | ||
|
|
||
| final class GoogleSheetLoaderTest extends FlowTestCase | ||
| { | ||
| public function test_load() : void | ||
| { | ||
| $httpContext = (new HttpClientContext()) | ||
| ->add( | ||
| new HttpRequestContext( | ||
| 'GET', | ||
| 'https://sheets.googleapis.com/v4/spreadsheets/1234567890', | ||
| ), | ||
| __DIR__ . '/../Fixtures/get-spreadsheet-response.json' | ||
| ) | ||
| ->add( | ||
| new HttpRequestContext( | ||
| 'PUT', | ||
| 'https://sheets.googleapis.com/v4/spreadsheets/1234567890/values/%27Sheet%27%21A1%3AB4?valueInputOption=USER_ENTERED', | ||
| '{"majorDimension":"ROWS","values":[["id","name"],[12345,"Norbert"],[54321,"Joseph"],[666,"Dominik"]]}', | ||
| ), | ||
| __DIR__ . '/../Fixtures/update-spreadsheet-response.json' | ||
| ); | ||
|
|
||
| $loader = to_google_sheet( | ||
| (new GoogleSheetsContext($httpContext)) | ||
| ->sheets(), | ||
| '1234567890', | ||
| 'Sheet', | ||
| ); | ||
|
|
||
| $loader->load( | ||
| rows( | ||
| row(int_entry('id', 12345), string_entry('name', 'Norbert')), | ||
| row(int_entry('id', 54321), string_entry('name', 'Joseph')), | ||
| row(int_entry('id', 666), string_entry('name', 'Dominik')) | ||
| ), | ||
| flow_context(config()) | ||
| ); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests\Integration; | ||
|
|
||
| use function Flow\ETL\Adapter\GoogleSheet\{from_google_sheet, | ||
| google_create_spreadsheet, | ||
| google_sheets, | ||
| to_google_sheet}; | ||
| use function Flow\ETL\DSL\{config, flow_context, int_entry, row, rows, str_entry, string_entry}; | ||
| use Flow\ETL\Tests\FlowTestCase; | ||
| use Google\Service\Sheets; | ||
| use Google\Service\Sheets\ClearValuesRequest; | ||
|
|
||
| final class GoogleSheetTest extends FlowTestCase | ||
| { | ||
| private const SHEET_NAME = 'Flow Sheet'; | ||
|
|
||
| private Sheets $service; | ||
|
|
||
| private string $spreadsheetId; | ||
|
|
||
| protected function setUp() : void | ||
| { | ||
| parent::setUp(); | ||
|
|
||
| $authFilename = __DIR__ . '/../../../../../../../auth.json.dist'; | ||
|
|
||
| if (!file_exists($authFilename)) { | ||
| self::markTestSkipped('auth.json.dist file is missing'); | ||
| } | ||
|
|
||
| try { | ||
| $this->service = google_sheets( | ||
| json_decode((string) file_get_contents($authFilename), true, 512, JSON_THROW_ON_ERROR), | ||
| Sheets::SPREADSHEETS | ||
| ); | ||
| } catch (\JsonException) { | ||
| self::markTestSkipped('auth.json.dist file contains not valid JSON'); | ||
| } | ||
|
|
||
| $spreadsheet = google_create_spreadsheet( | ||
| $this->service, | ||
| 'Flow test spreadsheet', | ||
| self::SHEET_NAME, | ||
| 'random@gmail.com' | ||
| ); | ||
|
|
||
| $this->spreadsheetId = $spreadsheet->spreadsheetId; | ||
|
|
||
| $this->clearTestSheet(); | ||
| } | ||
|
|
||
| protected function tearDown() : void | ||
| { | ||
| parent::tearDown(); | ||
|
|
||
| $this->clearTestSheet(); | ||
| } | ||
|
|
||
| public function test_load_and_extract() : void | ||
stloyd marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| { | ||
| $loader = to_google_sheet( | ||
| $this->service, | ||
| $this->spreadsheetId, | ||
| self::SHEET_NAME, | ||
| ); | ||
|
|
||
| $loader->load( | ||
| rows( | ||
| row(int_entry('id', 12345), string_entry('name', 'Norbert')), | ||
| row(int_entry('id', 54321), string_entry('name', 'Joseph')), | ||
| ), | ||
| $context = flow_context(config()) | ||
| ); | ||
|
|
||
| // Ensure previous rows are not overwritten | ||
| $loader->load( | ||
| rows( | ||
| row(int_entry('id', 666), string_entry('name', 'Dominik')) | ||
| ), | ||
| $context | ||
| ); | ||
stloyd marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| $extractor = from_google_sheet( | ||
| $this->service, | ||
| $this->spreadsheetId, | ||
| self::SHEET_NAME, | ||
| ); | ||
|
|
||
| $rowsArray = \iterator_to_array($extractor->extract(flow_context(config()))); | ||
| self::assertCount(3, $rowsArray); | ||
| self::assertSame(1, $rowsArray[0]->count()); | ||
| self::assertEquals(row(string_entry('id', '12345'), string_entry('name', 'Norbert')), $rowsArray[0]->first()); | ||
| self::assertSame(1, $rowsArray[1]->count()); | ||
| self::assertEquals(row(str_entry('id', '54321'), string_entry('name', 'Joseph')), $rowsArray[1]->first()); | ||
| self::assertSame(1, $rowsArray[2]->count()); | ||
| self::assertEquals(row(str_entry('id', '666'), string_entry('name', 'Dominik')), $rowsArray[2]->first()); | ||
| } | ||
|
|
||
| private function clearTestSheet() : void | ||
| { | ||
| $this->service->spreadsheets_values->clear( | ||
| $this->spreadsheetId, | ||
| "'" . self::SHEET_NAME . "'!A1:ZZ", | ||
| new ClearValuesRequest(), | ||
| ); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests\Unit; | ||
|
|
||
| use function Flow\ETL\DSL\{config, | ||
| date_entry, | ||
| datetime_entry, | ||
| flow_context, | ||
| int_entry, | ||
| row, | ||
| rows, | ||
| string_entry, | ||
| time_entry}; | ||
| use Flow\ETL\Adapter\GoogleSheet\{GoogleSheetLoader, ValueInputOption}; | ||
| use Google\Service\Sheets; | ||
| use Google\Service\Sheets\{GridProperties, Sheet, SheetProperties, Spreadsheet, ValueRange}; | ||
| use Google\Service\Sheets\Resource\{Spreadsheets, SpreadsheetsValues}; | ||
| use PHPUnit\Framework\TestCase; | ||
|
|
||
| final class GoogleSheetLoaderTest extends TestCase | ||
| { | ||
| public function test_load_with_entity_normalizer_options() : void | ||
| { | ||
| $sheetsMock = $this->createSheetsStub( | ||
| [ | ||
| ['date', 'datetime', 'time'], | ||
| ['03-05-2025', '2025-05-03 15:20', '01:00:00'], | ||
| ], | ||
| "'Sheet'!A1:C2" | ||
| ); | ||
| $loader = new GoogleSheetLoader( | ||
| $sheetsMock, | ||
| '1234567890', | ||
| 'Sheet' | ||
| ); | ||
| $loader->withDateTimeFormat('Y-m-d H:i'); | ||
| $loader->withDateFormat('d-m-Y'); | ||
| $loader->withInputOption(ValueInputOption::RAW); | ||
|
|
||
| $loader->load( | ||
| rows( | ||
| row( | ||
| date_entry('date', '2025-05-03 15:20:20'), | ||
| datetime_entry('datetime', '2025-05-03 15:20:20'), | ||
| time_entry('time', 'PT1H') | ||
| ), | ||
| ), | ||
| flow_context(config()) | ||
| ); | ||
| } | ||
|
|
||
| public function test_load_with_headers() : void | ||
| { | ||
| $sheetsMock = $this->createSheetsStub( | ||
| [ | ||
| ['id', 'name'], | ||
| [12345, 'Norbert'], | ||
| [54321, 'Joseph'], | ||
| ], | ||
| "'Sheet'!A1:B3" | ||
| ); | ||
| $loader = new GoogleSheetLoader( | ||
| $sheetsMock, | ||
| '1234567890', | ||
| 'Sheet', | ||
| ); | ||
| $loader->withInputOption(ValueInputOption::RAW); | ||
|
|
||
| $loader->load( | ||
| rows( | ||
| row(int_entry('id', 12345), string_entry('name', 'Norbert')), | ||
| row(int_entry('id', 54321), string_entry('name', 'Joseph')) | ||
| ), | ||
| flow_context(config()) | ||
| ); | ||
| } | ||
|
|
||
| public function test_load_without_headers() : void | ||
| { | ||
| $sheetsMock = $this->createSheetsStub( | ||
| [ | ||
| [12345, 'Norbert'], | ||
| [54321, 'Joseph'], | ||
| ], | ||
| "'Sheet'!A1:B2" | ||
| ); | ||
| $loader = new GoogleSheetLoader( | ||
| $sheetsMock, | ||
| '1234567890', | ||
| 'Sheet' | ||
| ); | ||
| $loader->withHeader(false); | ||
| $loader->withInputOption(ValueInputOption::RAW); | ||
|
|
||
| $loader->load( | ||
| rows( | ||
| row(int_entry('id', 12345), string_entry('name', 'Norbert')), | ||
| row(int_entry('id', 54321), string_entry('name', 'Joseph')) | ||
| ), | ||
| flow_context(config()) | ||
| ); | ||
| } | ||
|
|
||
| private function createSheetsStub(array $values, string $cellsRange) : Sheets | ||
| { | ||
| $gridProperties = new GridProperties(); | ||
| $gridProperties->rowCount = 1000; | ||
|
|
||
| $sheetProperties = new SheetProperties(); | ||
| $sheetProperties->title = 'Sheet'; | ||
| $sheetProperties->sheetId = 666; | ||
| $sheetProperties->setGridProperties($gridProperties); | ||
|
|
||
| $sheet = new Sheet(); | ||
| $sheet->setProperties($sheetProperties); | ||
|
|
||
| $spreadsheet = new Spreadsheet(); | ||
| $spreadsheet->setSpreadsheetId('1234567890'); | ||
| $spreadsheet->setSheets([$sheet]); | ||
|
|
||
| $spreadsheetsMock = $this->createMock(Spreadsheets::class); | ||
| $spreadsheetsMock->expects(self::once())->method('get')->with('1234567890')->willReturn($spreadsheet); | ||
|
|
||
| $sheets = new Sheets(); | ||
| $sheets->spreadsheets = $spreadsheetsMock; | ||
| $sheets->spreadsheets_values = $this->createMock(SpreadsheetsValues::class); | ||
| $sheets->spreadsheets_values->expects(self::once())->method('update')->with( | ||
| '1234567890', | ||
| $cellsRange, | ||
| new ValueRange( | ||
| [ | ||
| 'values' => $values, | ||
| 'majorDimension' => 'ROWS', | ||
| ], | ||
| ), | ||
| ['valueInputOption' => 'RAW'] | ||
| )->willReturn( | ||
| [ | ||
| 'spreadsheetId' => '1234567890', | ||
| 'updatedRange' => "'Sheet'!A1:B2", | ||
| 'updatedRows' => 2, | ||
| 'updatedColumns' => 2, | ||
| 'updatedCells' => 4, | ||
| ] | ||
| ); | ||
|
|
||
| return $sheets; | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests\Unit\RowsNormalizer; | ||
|
|
||
| use function Flow\ETL\DSL\{date_entry, | ||
| enum_entry, | ||
| json_entry, | ||
| list_entry, | ||
| time_entry, | ||
| type_list, | ||
| type_string, | ||
| uuid_entry}; | ||
| use Flow\ETL\Adapter\GoogleSheet\RowsNormalizer\EntryNormalizer; | ||
| use Flow\ETL\Adapter\GoogleSheet\ValueInputOption; | ||
| use Flow\ETL\Row\Entry; | ||
| use Flow\ETL\Row\Entry\{DateTimeEntry}; | ||
| use PHPUnit\Framework\TestCase; | ||
|
|
||
| final class EntryNormalizerTest extends TestCase | ||
| { | ||
| private EntryNormalizer $normalizer; | ||
|
|
||
| protected function setUp() : void | ||
| { | ||
| $this->normalizer = new EntryNormalizer( | ||
| 'Y-m-d H:i:s', | ||
| 'Y-m-d', | ||
| ); | ||
| } | ||
|
|
||
| public function test_normalize_date_entry() : void | ||
| { | ||
| $date = new \DateTimeImmutable('2024-03-15'); | ||
| $entry = date_entry('entry', $date); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals('2024-03-15', $result); | ||
| } | ||
|
|
||
| public function test_normalize_datetime_entry() : void | ||
| { | ||
| $dateTime = new \DateTimeImmutable('2024-03-15 14:30:00'); | ||
| $entry = new DateTimeEntry('entry', $dateTime); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals($dateTime->format('Y-m-d H:i:s'), $result); | ||
| } | ||
|
|
||
| public function test_normalize_default_entry() : void | ||
| { | ||
| $value = 'simple string'; | ||
| $entry = $this->createMock(Entry::class); | ||
| $entry->method('value')->willReturn($value); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals($value, $result); | ||
| } | ||
|
|
||
| public function test_normalize_enum_entry() : void | ||
| { | ||
| $enum = ValueInputOption::RAW; | ||
| $entry = enum_entry('entry', $enum); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals('RAW', $result); | ||
| } | ||
|
|
||
| public function test_normalize_json_entry() : void | ||
| { | ||
| $data = ['key' => 'value']; | ||
| $entry = json_entry('entry', $data); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals(json_encode($data), $result); | ||
| } | ||
|
|
||
| public function test_normalize_list_entry() : void | ||
| { | ||
| $list = ['item1', 'item2']; | ||
| $entry = list_entry('entry', $list, type_list(type_string())); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals(json_encode($list), $result); | ||
| } | ||
|
|
||
| public function test_normalize_null_value() : void | ||
| { | ||
| $entry = new DateTimeEntry('entry', null); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertSame('', $result); | ||
| } | ||
|
|
||
| public function test_normalize_time_entry() : void | ||
| { | ||
| $time = new \DateInterval('PT1H'); | ||
| $entry = time_entry('entry', $time); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals('01:00:00', $result); | ||
| } | ||
|
|
||
| public function test_normalize_uuid_entry() : void | ||
| { | ||
| $uuid = '550e8400-e29b-41d4-a716-446655440000'; | ||
| $entry = uuid_entry('entry', $uuid); | ||
|
|
||
| $result = $this->normalizer->normalize($entry); | ||
|
|
||
| self::assertEquals($uuid, $result); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests\Unit; | ||
|
|
||
| use Flow\ETL\Adapter\GoogleSheet\SheetAddress; | ||
| use PHPUnit\Framework\TestCase; | ||
|
|
||
| final class SheetAddressTest extends TestCase | ||
| { | ||
| public function test_calculating() : void | ||
| { | ||
| $range = SheetAddress::calculate(5, 10); | ||
|
|
||
| self::assertEquals('A1', $range->getStartCell()); | ||
| self::assertEquals('E10', $range->getEndCell()); | ||
| } | ||
|
|
||
| public function test_calculating_the_from_a_range() : void | ||
| { | ||
| $range = SheetAddress::calculate(2, 10, new SheetAddress('A1', 'B5', 'Sheet1')); | ||
|
|
||
| self::assertEquals('A1', $range->getStartCell()); | ||
| self::assertEquals('B10', $range->getEndCell()); | ||
| self::assertEquals('A', $range->getStartCellColumn()); | ||
| self::assertEquals(1, $range->getStartCellRow()); | ||
| self::assertEquals('B', $range->getEndCellColumn()); | ||
| self::assertEquals(10, $range->getEndCellRow()); | ||
| self::assertEquals('Sheet1', $range->getSheetName()); | ||
| } | ||
|
|
||
| public function test_with_multi_words_sheet_name() : void | ||
| { | ||
| $range = SheetAddress::calculate(2, 10, new SheetAddress('A1', 'B5', 'Sheet 1')); | ||
|
|
||
| self::assertEquals('Sheet 1', $range->getSheetName()); | ||
| self::assertEquals("'Sheet 1'!A1:B10", $range->toString()); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Flow\ETL\Adapter\GoogleSheet\Tests\Unit\Spreadsheet; | ||
|
|
||
| use Flow\ETL\Adapter\GoogleSheet\Spreadsheet\SpreadsheetManager; | ||
| use Flow\ETL\Exception\InvalidArgumentException; | ||
| use Flow\ETL\Tests\FlowTestCase; | ||
| use Google\Service\Sheets; | ||
| use Google\Service\Sheets\Resource\{Spreadsheets}; | ||
| use Google\Service\Sheets\{Spreadsheet}; | ||
|
|
||
| final class SpreadsheetManagerTest extends FlowTestCase | ||
| { | ||
| public function test_get_spreadsheet_properties_fails_without_sheet() : void | ||
| { | ||
| $this->expectException(InvalidArgumentException::class); | ||
| $this->expectExceptionMessage("Sheet 'Sheet' not found in spreadsheet '1234567890'"); | ||
|
|
||
| $manager = new SpreadsheetManager($this->createSheetsStub(), '1234567890'); | ||
| $manager->getSpreadsheetProperties('Sheet'); | ||
| } | ||
|
|
||
| private function createSheetsStub() : Sheets | ||
| { | ||
| $spreadsheet = new Spreadsheet(); | ||
| $spreadsheet->setSpreadsheetId('1234567890'); | ||
| $spreadsheet->setSheets([]); | ||
|
|
||
| $spreadsheetsMock = $this->createMock(Spreadsheets::class); | ||
| $spreadsheetsMock->expects(self::once())->method('get')->with('1234567890')->willReturn($spreadsheet); | ||
|
|
||
| $sheets = new Sheets(); | ||
| $sheets->spreadsheets = $spreadsheetsMock; | ||
|
|
||
| return $sheets; | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.