Repository: DivanteLtd/mage2vuestorefront
Branch: master
Commit: c96ec604b50d
Files: 71
Total size: 160.2 KB
Directory structure:
gitextract__jw74e5m/
├── .gitignore
├── CHANGELOG.md
├── LICENSE
├── README.md
├── doc/
│ ├── destination_es_product.json
│ ├── product_mapping.json
│ ├── source_magento_category.json
│ ├── source_magento_product.json
│ └── source_magento_review.json
├── docker-compose.yml
├── package.json
└── src/
├── adapters/
│ ├── abstract.js
│ ├── factory.js
│ ├── magento/
│ │ ├── abstract.js
│ │ ├── attribute.js
│ │ ├── cache_keys.js
│ │ ├── category.js
│ │ ├── cms_block.js
│ │ ├── cms_page.js
│ │ ├── magento2-rest-client/
│ │ │ ├── .npmignore
│ │ │ ├── LICENSE
│ │ │ ├── README.md
│ │ │ ├── index.js
│ │ │ ├── lib/
│ │ │ │ ├── attributes.js
│ │ │ │ ├── blocks.js
│ │ │ │ ├── bundle_options.js
│ │ │ │ ├── categories.js
│ │ │ │ ├── category_products.js
│ │ │ │ ├── configurable_children.js
│ │ │ │ ├── configurable_options.js
│ │ │ │ ├── custom_options.js
│ │ │ │ ├── log.js
│ │ │ │ ├── pages.js
│ │ │ │ ├── product_links.js
│ │ │ │ ├── product_media.js
│ │ │ │ ├── products.js
│ │ │ │ ├── rest_client.js
│ │ │ │ ├── reviews.js
│ │ │ │ ├── stock_items.js
│ │ │ │ ├── tax_rates.js
│ │ │ │ └── tax_rules.js
│ │ │ ├── magento2-rest-client.iml
│ │ │ ├── package.json
│ │ │ └── test/
│ │ │ ├── config.json
│ │ │ └── integration/
│ │ │ ├── categories.integration.test.js
│ │ │ ├── product_media.integration.test.js
│ │ │ └── products.integration.test.js
│ │ ├── product.js
│ │ ├── productcategories.js
│ │ ├── review.js
│ │ └── taxrule.js
│ └── nosql/
│ ├── abstract.js
│ └── elasticsearch.js
├── api/
│ └── routes/
│ └── magento.js
├── cli.js
├── config.js
├── helpers/
│ └── slugify.js
├── log.js
├── test_by_sku.sh
├── test_categoryextended.sh
├── test_fullreindex.sh
├── test_fullreindex_de.sh
├── test_fullreindex_it.sh
├── test_fullreindex_multiprocess.sh
├── test_multistore.sh
├── test_product.sh
├── test_product_delta.sh
├── test_product_msi.sh
├── test_product_worker.sh
├── tmp/
│ └── .gitignore
└── webapi.js
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
src/test_cleanup.sh
src/test_reindex.sh
src/node_modules
package-lock.json
node_modules
.idea
================================================
FILE: CHANGELOG.md
================================================
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.11.2] - 2020-11-12
### Fixed
- Enable indexing of review ratings - @soyamore (#107)
## [1.11.1] - 2020.09.15
### Added
- Updates magento2-rest-client dependecy from version 0.0.2 to 0.0.12 (#106)
- Bump lodash from 4.17.13 to 4.17.19 (#103)
## [1.11] - 2020.04.15
### Added
- Elastic7 support - @pkarw (#96)
- Add product attributes_metadata - @andrzejewsky (#99)
## [1.10] - 2019.07.10
### Added
- Added optional Redis Auth functionality. - @rain2o (#42)
- MSI support - @dimasch (#86)
### Fixed
- Import throwing an error when product's first category name was empty - @Loac-fr (#92)
- Typos in documentation - @kkdg, @adityasharma7 (#90, #91)
## [1.9] - 2019.03.14
### Added
- New ENV variable `SEO_USE_URL_DISPATCHER` (default = true) added. When set, then the `product.url_path` and `category.url_path` are automatically populated for the UrlDispatche featu$
## [1.8.1] - 2019.02.13
### Changed / improved
- `elasticsearch.apiVesion` with default = 5.6 added to the config
## [1.8.0] - 2019.02.08
### Added
- Video data mapper @rain2go [#75](https://github.com/DivanteLtd/mage2vuestorefront/pull/75)
## [1.8.0]
### Added
- Setting `configurable_options.label` from the attribute meta descriptor. **Note:** When You modify any configurable attribute label in Magento You should reindex all products now
- Configurable parent refresh sync - enabled in the `productsdelta` and `productsworker` modes and in `products --sku=<singleSku>`. This mode is refreshing the configurable parent product for the simple child which requires update. Its' required to start the `clis.js productsworker` (example call: `test_product_worker.sh`) for processing these parent updates,
- Example calls added: `test_product_delta.sh` - for delta indexer, `test_product_worker.sh` for products worker, `test_fullreindex_multiprocess.sh` for multi process/parallel updates, `test_by_sky.sh` - for single SKU updates.
### Removed
- Mongodb support has been removed
- old `package.json` and `yarn.lock` from `src` directory
## [1.7.1] - 2019-01-29
### Added
- This changelog file
### Fixed
- Slugify funtion properly parse regional characters changing them to latin
- yarn.lock file rebuild after some issues with merging
### Changed
- Slugify funtion moved outside adapter and exposed as helper
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2017 Divante Ltd.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# mage2alokai
### Stay connected
[](https://github.com/vuestorefront/vue-storefront)
[](https://twitter.com/vuestorefront)
[](https://www.youtube.com/c/VueStorefront)
[](https://discord.vuestorefront.io)
For those who would love to work with Magento on backend but use NoSQL power on the frontend. Two way / real time data synchronizer.
It's part of [alokai project - first Progressive Web App for eCommerce](https://github.com/DivanteLtd/vue-storefront) with Magento2 support.
Some details about the rationale and our goals [here](https://www.linkedin.com/pulse/magento2-nosql-database-pwa-support-piotr-karwatka)
It synchronizes all the products, attributes, taxrules, categories and links between products and categories.
This is multi-process data synchronizer between Magento to Alokai ElasticSearch database.
At this point synchronization works with following entities:
- Products
- Categories
- Taxrules
- Attributes
- Product-to-categories
- Reviews (require custom module Divante/ReviewApi to work)
- Cms Blocks & Pages (require custom module [SnowdogApps/magento2-cms-api](https://github.com/SnowdogApps/magento2-cms-api))
Categories and Product-to-categories links are additionaly stored in Redis cache for rapid-requests (for example from your WebAPI). Our other project [alokai-api](https://github.com/DivanteLtd/vue-storefront-api) exposes this database to be used in PWA/JS webapps.
Datasync uses oauth + magento2 rest API to get the data.
KUE is used for job queueing and multi-process/multi-tenant processing is enabled by default
ElasticSearch is used for NoSQL database
Redis is used for KUE queue backend
By default all services are used without authorization and on default ports (check out config.js or ENV variables for change of this behavior).
**Tutorial on installation / integration [manual for Alokai connectivity](https://medium.com/@piotrkarwatka/vue-storefront-how-to-install-and-integrate-with-magento2-227767dd65b2)**
## How to perform full / initial import for Alokai
To get started with VS we must start with some very basics about the architecture; the project is backed by three separate Node.js applications
### Alokai Architecture
[alokai (Github)](https://github.com/DivanteLtd/vue-storefront) — is the main project where you can also find most of the documentation, issues mapped to further releases and other resources to start with — Vue.js on webpack.
[alokai-api (Github)](https://github.com/DivanteLtd/vue-storefront-api) — is the API layer which provides the data to vue-storefront app — Node.js, Express; This project consist of docker instances for Redis and ElasticSearch required by mage2vuestorefront and pimcore2vuestorefront
mage2alokai — THIS project -data bridges which are in charge of moving data back from Magento2 to Alokai data store.
You must install `alokai-api` locally. You may install it using the Alokai installer - see details. Or manually by executing the sequence of commands:
```bash
git clone https://github.com/DivanteLtd/vue-storefront-api
cd vue-storefront-api
npm install
npm run migrate
docker-compose up -d
npm run dev
```
The key command is `docker-compose up -d` which runs the ElasticSearch and Redis instances - both required by `mage2vuestorefront`
### Elastic 7 Support
By default, Alokai API docker files and config are based on Elastic 5.6. We plan to change the default Elastic version to 7 with the 1.11 stable release. As for now, the [Elastic 7 support](https://github.com/DivanteLtd/vue-storefront-api/pull/342) is marked as **experimental**.
In order to index data to Elastic 7 please make sure you set the proper `apiVersion` in the `config.js`:
```js
elasticsearch: {
apiVersion: process.env.ELASTICSEARCH_API_VERSION || '7.1'
},
```
or just use the env variable:
```bash
export ELASTICSEARCH_API_VERSION=7.1
```
Starting from [Elasitc 6 and 7](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html) we can have **just single** document type per single index. Alokai used to have `product`, `category` ... types defined in the `vue_storefront_catalog`.
From now on, we're using the separate indexes per each entity type. The convention is: `${indexName}_${entityType}`. If your' **logical index name** is `vue_storefront_catalog` then it will be mapped to the **physical indexes** of: `vue_storefront_catalog_product`, `vue_storefront_catalog_category` ...
### Initial Alokai import
Now, You're ready to run the importer. Please check the [config file](https://github.com/DivanteLtd/mage2vuestorefront/blob/master/src/config.js). You may setup the Magento access data and URLs by config values or ENV variables.
We'll use in the following example - the ENV variables. The simplest command sequence to perform full reindex is:
```bash
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
export INDEX_NAME=vue_storefront_catalog
node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
node --harmony cli.js reviews
```
After installing the 3rd party Magneto module ([SnowdogApps/magento2-cms-api](https://github.com/SnowdogApps/magento2-cms-api)) there are two additional imports available:
```
node --harmony cli.js blocks
node --harmony cli.js pages
```
**Please note:**
- `--removeNonExistent` option means - all records that were found in the index but currently don't exist in the API feed - will be removed. Please use this option ONLY for the full reindex!
- `INDEX_NAME` by default is set to the `vue_storefront_catalog` but You may set it to any other elastic search index name.
- The `categories` importer option `--generateUniqueUrlKeys` is by default set to true. This is due the fact that in Magento2, the `category.url_key` field is not mandatory unique and from v. 1.7 Alokai uses the `category.url_key` to display the category details without any client's side modification.
- `PRODUCTS_EXCLUDE_DISABLED` by default is set to `false`. To only import enabled products set this to `true`.
**Cache invalidation:** Recent version of Alokai do support output caching. Output cache is being tagged with the product and categories id (products and categories used on specific page). Mage2vuestorefront can invalidate cache of product and category pages if You set the following ENV variables:
```bash
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
```
- `VS_INVALIDATE_CACHE_URL` is a cache to the Alokai instance - used as a webhook to clear the output cache.
Please note:
After data import - especially when You're not sure about the product attributes data types - please **reindex** ElasticSearch to establish the correct / current database schema. You may do this using [Database tool](https://github.com/DivanteLtd/vue-storefront/blob/master/doc/Database%20tool.md) in the `vue-storefront-api` folder:
```bash
cd vue-storefront-api
npm run db rebuild -- --indexName=vue_storefront_catalog
```
If You like to create a new, empty index please run:
```bash
cd vue-storefront-api
npm run db new -- --indexName=vue_storefront_catalog
```
### Checking indexed data
If you want to see how many products were stored into Elastic data store, you can use Kibana to do so. Kibana is part of alokai-api. Once you start docker containers of alokai-api you can access it on http://localhost:5601/.
To see count of indexed products go to DEV tools and run following query:
```
GET vue_storefront_catalog/product/_count
```
See https://www.elastic.co/guide/en/kibana/current/console-kibana.html to find out more.
### Delta indexer
After initial setup and full-reindex You may want to add indexer to the `crontab` to index only modified product records.
This is fairly easy - You just need to add the following command to crontab:
```bash
node --harmony cli.js productsdelta --partitions=1
```
This command will execute full reindex at first call - and then will be storing the last index date in the `.lastIndex.json` and downloading only these products which have `updated_at` > last index date.
If you have a multistore setup and would like to use the delta indexer for each storeview you can not use the delta timestamp from `.lastIndex.json` for all stores; instead
you will need to set the `INDEX_META_PATH` to a unique value for each store you are indexing. For instance:
```
export INDEX_META_PATH=.lastIndex-UK.json && node --harmony cli.js productsdelta --partitions=1
```
Please note: Magento2 has a bug with altering `updated_at` field. Please install [a fix for that](https://github.com/codepeak/magento2-productfix) before using this method:
```bash
composer require codepeak/magento2-productfix
php bin/magento cache:flush
```
### Parent products updates
Please note if there is a `simple` product update request coming from Delta Indexer or On Demand indexer `mage2vuestorefront` will - by default - check and update the `configurable`/parent product as well. The parent product update is scheduled in the `productsworker` mode - using KUE queue. Please make sure You're runinng at least one worker instance to process these on-demand request:
```bash
cd mage2vuestorefront/src
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export PORT=6060
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js productsworker
```
This second process will take care of indexing the parent products updates whetever any of it's `configurable_children` simple products has been changed.
### On-demand indexer (experimental!)
Mage2nosql supports an on-demand indexer - where Magento calls a special webhook to update modified products.
In the current version the webhook notifies mage2vuestorefront about changed product SKUs and mage2vuestorefront pulls the modified products data via Magento2 APIs.
First. You should install [Magento2 module called VsBridge](https://github.com/DivanteLtd/magento2-vsbridge)
Second. Deploy mage2vuestorefront on the server.
Then You may want to start a webapi process:
```bash
cd mage2vuestorefront/src
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export PORT=6060
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony webapi.js
```
The API will be listening on port 6060. Typically non-standard ports like this one are not exposed on the firewall. Please consider setting up [simple nginx proxy for this service](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-16-04).
Anyway - this API must be publicly available via Internet OR You must have the mage2vuestorefront installed on the same machine like Magento2.
Go to Your Magento2 admin panel, then to Stores -> Configuration -> VsBridge and set-up "Edit product" url to: `http://localhost:6060/magento/products/update`. **Please note:** Product delete endpoint hasn't been implemented yet and it's good chance for Your PR.
After having the webapi up and runing and this endpoint set, any Product save action will call `POST http://localhost:6060/magento/products/update` with the body set to `{"sku": ["modified-sku-list"]}`.
Webapi will [add the products to the queue](https://github.com/DivanteLtd/mage2vuestorefront/blob/b44e7ede9aeb27f308e2a87033251a2491640da8/src/api/routes/magento.js#L19).
Please run the queue worker to process all the queued updates (You may run multiple queue workers even distributed across many machines):
```bash
cd mage2vuestorefront/src
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export PORT=6060
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js productsworker
```
**Please note:** We're using [kue based on Redis queue](https://github.com/Automattic/kue) which may be configured via `src/config.js` - `kue` + `redis` section.
**Please note:** Redis now supports auth. In order to use Redis with auth simply pass the password to the `REDIS_AUTH` env variable.
### Multistore setup
Multiwebsite support starts with the ElasticSearch indexing. Basically - each store has it's own ElasticSearch index and should be populated separately using [mage2vuestorefront](https://github.com/DivanteLtd/mage2vuestorefront) tool.
The simplest script to index multi site:
```bash
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'German store - de'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/de
export INDEX_NAME=vue_storefront_catalog_de
node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
echo 'Italian store - it'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/it
export INDEX_NAME=vue_storefront_catalog_it
node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
export INDEX_NAME=vue_storefront_catalog
node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
```
As You may see it's just a **it** or **de** store code which is added to the base Magento2 REST API urls that makes the difference and then the **INDEX_NAME** set to the dedicated index name.
In the result You should get:
- *vue_storefront_catalog_it* - populated with the "it" store data
- *vue_storefront_catalog_de* - populated with the "it" store data
- *vue_storefront_catalog* - populated with the "default" store data
Then, to use these indexes in the Alokai You should index the database schema using the `alokai-api` db tool:
```bash
npm run db rebuild -- --indexName=vue_storefront_catalog_it
npm run db rebuild -- --indexName=vue_storefront_catalog_de
npm run db rebuild -- --indexName=vue_storefront_catalog
```
More on <a href="https://github.com/DivanteLtd/vue-storefront/blob/master/doc/Multistore%20setup.md">how to setup Alokai in the Multistore mode</a>.
### Indexing configurable products attributes for filters
If You like to have Category filter working with configurable products - You need to expand the `product.configurable_children.attrName` to `product.attrName_options` array. This is automatically done by [mage2vuestorefront](https://github.com/DivanteLtd/mage2vuestorefront) for all attributes set as `product.configurable_options` (by default: color, size). If You like to add additional fields like `manufacturer` to the filters You need to expand `product.manufacturer_options` field. The easiest way to do so is to set `config.product.expandConfigurableFilters` to `['manufacturer']` and re-run the `mage2vuestorefront` indexer.
## FAQ
Here You can find some frequently asked questions answered:
### I've been playing with Alokai for quite a while now and now I see that my catalog rule (-20% on all products) is not applied in one shop.
Please make sure that You've got the `config.synchronizeCatalogSpecialPrices` (env: `PRODUCTS_SPECIAL_PRICES`) and `config.renderCatalogRegularPrices` (env: `PRODUCTS_RENDER_PRICES`) set to `true` (default is `false`). Otherwise only the catalog prices will be synced (without dynamic pricing rules applied). You can also use the Alokai [dynamic-pricing option](https://divanteltd.github.io/vue-storefront/guide/integrations/direct-prices-sync.html) for the same purpose.
## Advanced usage
Start Elasticsearch and Redis:
- `docker-compose up`
Install:
- `npm install`
- `cd src`
Config - see: config.js or use following ENV variables:
- `MAGENTO_URL`
- `MAGENTO_CONSUMER_KEY`
- `MAGENTO_CONSUMER_SECRET`
- `MAGENTO_ACCESS_TOKEN`
- `MAGENTO_ACCESS_TOKEN_SECRET`
- `DATABASE_URL` (default: 'elasticsearch://localhost:9200/vue_storefront_catalog')
Run:
- `cd src/` and then:
- `node --harmony cli.js fullreindex` - synchronizes all the categories, products and links between products and categories
Other commands supported:
- `node --harmony cli.js products --partitions=10`
- `node --harmony cli.js products --partitions=10 --initQueue=false` - run the products sync worker (product sync jobs should be populated eslewhere - it's used to run multi-tenant environment of workers)
- `node --harmony cli.js products --partitions=10 --delta=true` - check products changed since last run; compared by updated_at field
- `node --harmony cli.js productcategories` - to synchronize the links between products and categories it *should be run before* products synchronization because it populates Redis cache assigments for product-to-category link
- `node --harmony cli.js categories`
- `node --harmony cli.js products --adapter=magento --partitions=1 --skus=24-WG082-blue,24-WG082-pink` - to pull out only selected SKUs
- `node --harmony cli.js productsworker --adapter=magento --partitions=10` - run queue worker for pulling out individual products (jobs can be assigned by webapi.js microservice triggers; it can be called by webhook for example from within Magento2 plugin)
- `node --harmony webapi.js` - run localhost:3000 service endpoint for adding queue tasks
WebAPI:
- `node --harmony webapi.js`
- `curl localhost:8080/api/magento/products/pull/WT09-XS-Purple` - to schedule data refresh for SKU=WT09-XS-Purple
- `node --harmony cli.js productsworker` - to run pull request processor
Available options:
- `partitions=10` - number of concurent processes, by default number of CPUs core given
- `adapter=magento` - for now only Magento is supported
- `delta` - sync products changed from last run
- command names: `products` / `attributes` / `taxrule` / `categories` / `productsworker` / `productcategories` / `productsdelta`
================================================
FILE: doc/destination_es_product.json
================================================
"entity_id": "1351",
"attribute_set_id": "10",
"type_id": "simple",
"sku": "6343908",
"has_options": "0",
"required_options": "0",
"created_at": "2017-04-06 16:25:59",
"updated_at": "2017-04-06 16:25:59",
"visibility": "4",
"price": [
{
"price": "32.5000",
"original_price": "34.9000",
"is_discount": true,
"customer_group_id": "0"
},
{
"price": "32.5000",
"original_price": "34.9000",
"is_discount": true,
"customer_group_id": "1"
},
{
"price": "32.5000",
"original_price": "34.9000",
"is_discount": true,
"customer_group_id": "4"
},
{
"price": "32.5000",
"original_price": "34.9000",
"is_discount": true,
"customer_group_id": "5"
}
],
"category": [
{
"category_id": 2
},
{
"category_id": 4,
"is_parent": true,
"name": "Książki"
},
{
"category_id": 26,
"is_parent": true,
"name": "biznes"
},
{
"category_id": 232,
"is_parent": true,
"name": " praca"
},
{
"category_id": 3101,
"name": "GRUPY"
},
{
"category_id": 3220,
"is_parent": true,
"name": " Donald Trump. 45. Prezydent Stanów Zjednoczonych "
}
],
"name": [
"Sukces mimo wszystko. Donald Trump"
],
"image": [
"/9/9/99906343908.jpg"
],
"ean": [
"9788328333772"
],
"pkwiu": [
"58.11"
],
"availability": [
"7"
],
"isbn": [
"978-83-283-3377-2"
],
"authors_es": [
"_68450_,_68449_,_68450_,_68449_"
],
"publishers_es": [
"_62556_,_62556_,_62556_,_62556_"
],
"producers_es": [
"_62556_,_62556_,_62556_,_62556_"
],
"status": [
1
],
"option_text_status": [
"Enabled"
],
"tax_class_id": [
2
],
"option_text_tax_class_id": [
"Taxable Goods"
],
"description": [
"<h3>Ucz się od mistrza, a będziesz mieć świat u stóp!</h3> Kto powiedział, że cokolwiek na świecie jest poza Twoim zasięgiem? Pewnie nie zostaniesz już ani pierwszym kosmonautą, ani odkrywcą Ameryki, ale w gruncie rzeczy możesz osiągnąć niemal wszystko! Kluczem do sukcesu — Twojego i wszystkich innych ludzi — jest edukacja! Badania i wiedza, eksperymentowanie i doświadczanie: to wszystko stoi za sukcesem <strong>Donalda Trumpa</strong>. To dzięki nim ten ekscentryczny miliarder odnosi sukcesy we wszystkim, czego się dotknie. To nie magia — to twarde dane, chłodne analizy, podstawowe zasady relacji biznesowych i społecznych.<br /> <br /> W tej książce <strong>Donald Trump</strong> pokazuje nam, skąd sam czerpie wiedzę i jak wykorzystuje ją na co dzień. Dzieli się swoimi przemyśleniami, odpowiada na pytania zadawane mu na blogu i daje wskazówki swoim ewentualnym naśladowcom. Jeśli chcesz wspiąć się na sam szczyt ekonomicznej (lub społecznej) piramidy, jeśli nie boisz się wyzwań i ciężkiej pracy, jesteś głodny sukcesu i gotów na wiele wyrzeczeń, sprawdź, jak może Ci w tym pomóc <strong>Donald Trump</strong>. Zobacz, co naprawdę napędza świat biznesu, jak nakreślić skuteczny plan działania i nieustępliwie walczyć o swoje. Odkryj w sobie pasję i pozytywną motywację, stale poszerzaj horyzonty i nikomu nie daj się zastraszyć ani zepchnąć z obranej drogi. A przede wszystkim rób to, co Ci sprawia przyjemność, co stanowi Twoją prawdziwą pasję, co budzi w Tobie dreszcz podniecenia. Podążaj za wskazówkami nowego prezydenta Stanów Zjednoczonych, a świat stanie przed Tobą otworem! <ul> <li>Zawsze mierz wysoko.</li> <li>Stale dąż do perfekcji.</li> <li>Nie lekceważ przeczuć.</li> <li>Stawiaj na aktywność.</li> <li>Wykonuj pracę, którą kochasz.</li> </ul> <h3>Odważni ludzie wyrastają ponad szklany sufit!</h3> <hr /><strong>Donald J. Trump</strong> — człowiek, którego właściwie nie trzeba nikomu przedstawiać. Krezus i ekscentryk, prezes Organizacji Trumpa, nowy prezydent Stanów Zjednoczonych. Z zawodu inwestor nieruchomości i autor bestsellerów, z zamiłowania filantrop. "
],
"short_description": [
"Ucz się od mistrza, a będziesz mieć świat u stóp! Kto powiedział, że cokolwiek na świecie jest poza Twoim zasięgiem? Pewnie nie zostaniesz już ani pierwszym kosmonautą, ani odkrywcą Ameryki, ale w gruncie rzeczy możesz osiągnąć niemal wszystko!..."
],
"special_price": [
32.5
],
"stock": {
"is_in_stock": true,
"qty": 0
}
},
"sort": [
"59",
"59"
]
}
================================================
FILE: doc/product_mapping.json
================================================
{
"vue_storefront_catalog": {
"mappings": {
"product": {
"properties": {
"actors_es": {
"type": "string"
},
"attribute_set_id": {
"type": "string"
},
"authors_es": {
"type": "string"
},
"availability": {
"type": "string"
},
"category": {
"properties": {
"category_id": {
"type": "long"
},
"is_parent": {
"type": "boolean"
},
"is_virtual": {
"type": "string"
},
"name": {
"type": "string"
},
"position": {
"type": "long"
}
}
},
"composers_es": {
"type": "string"
},
"created_at": {
"type": "string"
},
"description": {
"type": "string"
},
"directors_es": {
"type": "string"
},
"ean": {
"type": "string"
},
"entity_id": {
"type": "string"
},
"has_options": {
"type": "string"
},
"image": {
"type": "string"
},
"is_new": {
"type": "long"
},
"is_preorder": {
"type": "long"
},
"isbn": {
"type": "string"
},
"name": {
"type": "string"
},
"option_text_is_new": {
"type": "string"
},
"option_text_is_preorder": {
"type": "string"
},
"option_text_status": {
"type": "string"
},
"option_text_tax_class_id": {
"type": "string"
},
"performers_es": {
"type": "string"
},
"pkwiu": {
"type": "string"
},
"price": {
"properties": {
"customer_group_id": {
"type": "string"
},
"is_discount": {
"type": "boolean"
},
"original_price": {
"type": "string"
},
"price": {
"type": "string"
}
}
},
"producers_es": {
"type": "string"
},
"publishers_es": {
"type": "string"
},
"required_options": {
"type": "string"
},
"short_description": {
"type": "string"
},
"sku": {
"type": "string"
},
"special_price": {
"type": "double"
},
"status": {
"type": "long"
},
"stock": {
"properties": {
"is_in_stock": {
"type": "boolean"
},
"qty": {
"type": "long"
}
}
},
"tax_class_id": {
"type": "long"
},
"type_id": {
"type": "string"
},
"updated_at": {
"type": "string"
},
"visibility": {
"type": "string"
}
}
}
}
}
}
================================================
FILE: doc/source_magento_category.json
================================================
{
"id": 0,
"parent_id": 0,
"name": "string",
"is_active": true,
"position": 0,
"level": 0,
"product_count": 0,
"children_data": [
{}
]
}
================================================
FILE: doc/source_magento_product.json
================================================
ModelModel Schema
{
"items": [
{
"id": 0,
"sku": "string",
"name": "string",
"attribute_set_id": 0,
"price": 0,
"status": 0,
"visibility": 0,
"type_id": "string",
"created_at": "string",
"updated_at": "string",
"weight": 0,
"extension_attributes": {
"stock_item": {
"item_id": 0,
"product_id": 0,
"stock_id": 0,
"qty": 0,
"is_in_stock": true,
"is_qty_decimal": true,
"show_default_notification_message": true,
"use_config_min_qty": true,
"min_qty": 0,
"use_config_min_sale_qty": 0,
"min_sale_qty": 0,
"use_config_max_sale_qty": true,
"max_sale_qty": 0,
"use_config_backorders": true,
"backorders": 0,
"use_config_notify_stock_qty": true,
"notify_stock_qty": 0,
"use_config_qty_increments": true,
"qty_increments": 0,
"use_config_enable_qty_inc": true,
"enable_qty_increments": true,
"use_config_manage_stock": true,
"manage_stock": true,
"low_stock_date": "string",
"is_decimal_divided": true,
"stock_status_changed_auto": 0,
"extension_attributes": {}
},
"bundle_product_options": [
{
"option_id": 0,
"title": "string",
"required": true,
"type": "string",
"position": 0,
"sku": "string",
"product_links": [
{
"id": "string",
"sku": "string",
"option_id": 0,
"qty": 0,
"position": 0,
"is_default": true,
"price": 0,
"price_type": 0,
"can_change_quantity": 0,
"extension_attributes": {}
}
],
"extension_attributes": {}
}
],
"downloadable_product_links": [
{
"id": 0,
"title": "string",
"sort_order": 0,
"is_shareable": 0,
"price": 0,
"number_of_downloads": 0,
"link_type": "string",
"link_file": "string",
"link_file_content": {
"file_data": "string",
"name": "string",
"extension_attributes": {}
},
"link_url": "string",
"sample_type": "string",
"sample_file": "string",
"sample_file_content": {
"file_data": "string",
"name": "string",
"extension_attributes": {}
},
"sample_url": "string",
"extension_attributes": {}
}
],
"downloadable_product_samples": [
{
"id": 0,
"title": "string",
"sort_order": 0,
"sample_type": "string",
"sample_file": "string",
"sample_file_content": {
"file_data": "string",
"name": "string",
"extension_attributes": {}
},
"sample_url": "string",
"extension_attributes": {}
}
],
"giftcard_amounts": [
{
"attribute_id": 0,
"website_id": 0,
"value": 0,
"website_value": 0,
"extension_attributes": {}
}
],
"configurable_product_options": [
{
"id": 0,
"attribute_id": "string",
"label": "string",
"position": 0,
"is_use_default": true,
"values": [
{
"value_index": 0,
"extension_attributes": {}
}
],
"extension_attributes": {},
"product_id": 0
}
],
"configurable_product_links": [
0
]
},
"product_links": [
{
"sku": "string",
"link_type": "string",
"linked_product_sku": "string",
"linked_product_type": "string",
"position": 0,
"extension_attributes": {
"qty": 0
}
}
],
"options": [
{
"product_sku": "string",
"option_id": 0,
"title": "string",
"type": "string",
"sort_order": 0,
"is_require": true,
"price": 0,
"price_type": "string",
"sku": "string",
"file_extension": "string",
"max_characters": 0,
"image_size_x": 0,
"image_size_y": 0,
"values": [
{
"title": "string",
"sort_order": 0,
"price": 0,
"price_type": "string",
"sku": "string",
"option_type_id": 0
}
],
"extension_attributes": {}
}
],
"media_gallery_entries": [
{
"id": 0,
"media_type": "string",
"label": "string",
"position": 0,
"disabled": true,
"types": [
"string"
],
"file": "string",
"content": {
"base64_encoded_data": "string",
"type": "string",
"name": "string"
},
"extension_attributes": {
"video_content": {
"media_type": "string",
"video_provider": "string",
"video_url": "string",
"video_title": "string",
"video_description": "string",
"video_metadata": "string"
}
}
}
],
"tier_prices": [
{
"customer_group_id": 0,
"qty": 0,
"value": 0,
"extension_attributes": {}
}
],
"custom_attributes": [
{
"attribute_code": "string",
"value": "string"
}
]
}
],
"search_criteria": {
"filter_groups": [
{
"filters": [
{
"field": "string",
"value": "string",
"condition_type": "string"
}
]
}
],
"sort_orders": [
{
"field": "string",
"direction": "string"
}
],
"page_size": 0,
"current_page": 0
},
"total_count": 0
}
================================================
FILE: doc/source_magento_review.json
================================================
{
"id": 0,
"title": "string",
"detail": "string",
"nickname": "string",
"ratings": [{
"id": 0,
"review_id": 0,
"attribute_code": "string",
"value": 0
}],
"customer_id": null,
"review_entity": "product",
"review_type": 0,
"review_status": 0,
"created_at": "string",
"entity_pk_value": 0,
"stores": []
}
================================================
FILE: docker-compose.yml
================================================
version: '2'
services:
esm1:
image: elasticsearch:7.3.2
container_name: esm1
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
volumes:
- esdat1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- esnet
esm2:
image: elasticsearch:7.3.2
container_name: esm2
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "discovery.zen.ping.unicast.hosts=esm1"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
volumes:
- esdat2:/usr/share/elasticsearch/data
networks:
- esnet
redis:
image: redis
ports:
- "6379:6379"
volumes:
esdat1:
driver: local
esdat2:
driver: local
networks:
esnet:
================================================
FILE: package.json
================================================
{
"name": "mage2vuestorefront",
"private": true,
"version": "1.11.1",
"description": "Magento sync for products, categories, users and orders",
"author": "Piotr Karwatka",
"license": "MIT",
"main": "src/cli.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"dependencies": {
"@elastic/elasticsearch": "^7.3.0",
"agentkeepalive": "^3.3.0",
"body-parser": "^1.17.1",
"commander": "^2.18.0",
"elasticsearch-deletebyquery": "^1.6.0",
"express": "^4.15.2",
"jsonfile": "^4.0.0",
"jwt-simple": "^0.5.1",
"kue": "^0.11.5",
"lodash": "^4.17.19",
"magento2-rest-client": "https://github.com/DivanteLtd/magento2-rest-client",
"moment": "^2.22.2",
"node-sync": "^0.2.1",
"passport": "^0.3.2",
"passport-jwt": "^4.0.0",
"promise-queue": "^2.2.3",
"queue": "^4.4.0",
"redis-jsonify": "^1.0.1",
"remove-accents": "^0.4.2",
"request": "^2.88.0",
"sha1": "^1.1.1"
},
"devDependencies": {}
}
================================================
FILE: src/adapters/abstract.js
================================================
'use strict';
const AdapterFactory = require('./factory');
const Redis = require('redis');
class AbstractAdapter {
validateConfig(config) {
if (!config['db']['url'])
throw Error('db.url must be set up in config');
}
constructor(app_config) {
this.config = app_config;
let factory = new AdapterFactory(app_config);
this.db = factory.getAdapter('nosql', app_config.db.driver);
if (global.cache == null) {
this.cache = Redis.createClient(this.config.redis); // redis client
this.cache.on('error', (err) => { // workaround for https://github.com/NodeRedis/node_redis/issues/713
this.cache = Redis.createClient(this.config.redis); // redis client
});
// redis auth if provided
if (this.config.redis.auth) {
this.cache.auth(this.config.redis.auth);
}
global.cache = this.cache;
} else this.cache = global.cache;
this.update_document = true; // should we update database with new data from API? @see productcategory where this is disabled
this.total_count = 0;
this.page_count = 0;
this.page_size = 50;
this.page = 1;
this.current_context = {};
this.use_paging = false;
this.is_federated = false;
this.validateConfig(this.config);
this.tasks_count = 0;
}
isValidFor(entity_type) {
throw Error('isValidFor must be implemented in specific class');
}
getCurrentContext() {
return this.current_context;
}
/**
* Default done callback called after all main items are processed by processItems
*/
defaultDoneCallback() {
return;
}
/**
* Run products/categories/ ... import
* @param {Object} context import context with parameter such "page", "size" and other search parameters
*/
run(context) {
this.current_context = context;
if (!(this.current_context.transaction_key))
this.current_context.transaction_key = new Date().getTime(); // the key used to filter out records NOT ADDED by this import
this.db.connect(() => {
logger.info('Connected correctly to server');
logger.info(`TRANSACTION KEY = ${this.current_context.transaction_key}`);
this.onDone = this.current_context.done_callback ? (
() => {
this.defaultDoneCallback();
this.current_context.done_callback();
}
): this.defaultDoneCallback;
let exitCallback = this.onDone;
this.getSourceData(this.current_context)
.then(this.processItems.bind(this))
.catch((err) => {
logger.error(err);
exitCallback();
});
});
}
/**
* Implement some item related operations - executed BEFORE saving to the database
* @param {Object} item
*/
preProcessItem(item) {
return new Promise((done, reject) => { done(); });
}
/**
* Remove records from database other than specific transaction_key
* @param {int} transaction_key
*/
cleanUp(transaction_key) {
this.db.connect(() => {
logger.info(`Cleaning up with tsk = ${transaction_key}`);
this.db.cleanupByTransactionkey(this.getCollectionName(), transaction_key);
});
}
prepareItems(items) {
if(!items)
return items;
if (items.total_count)
this.total_count = items.total_count;
if (!Array.isArray(items))
items = new Array(items);
return items;
}
isFederated() {
return this.is_federated;
}
processItems(items, level) {
if (isNaN(level))
level = 0;
items = this.prepareItems(items);
if (!items) {
logger.error('No items given to processItems call!');
return;
}
let count = items.length;
let index = 0;
if (count == 0) {
logger.warn('No records to process!');
return this.onDone(this);
} else
this.tasks_count += count;
let db = this.db;
if (!db)
throw new Error('No db adapter connection established!');
if (this.total_count)
logger.info(`Total count is: ${this.total_count}`)
items.map((item) => {
this.preProcessItem(item).then((item) => {
this.tasks_count--;
item.tsk = this.getCurrentContext().transaction_key; // transaction key for items that can be then cleaned up
logger.info(`Importing ${index} of ${count} - ${this.getLabel(item)} with tsk = ${item.tsk}`);
logger.info(`Tasks count = ${this.tasks_count}`);
if (this.update_document)
this.db.updateDocument(this.getCollectionName(), this.normalizeDocumentFormat(item))
else
logger.debug('Skipping database update');
if (item.children_data && item.children_data.length > 0) {
logger.info(`--L:${level} Processing child items ...`);
this.processItems(item.children_data, level + 1);
}
if (this.tasks_count == 0 && !this.use_paging) { // this is the last item!
logger.info('No tasks to process. All records processed!');
this.db.close();
return this.onDone(this);
} else {
if (index == (count - 1)) { // page done!
logger.debug(`--L:${level} Level done! Current page: ${this.page} of ${this.page_count}`);
if (parseInt(level) == 0) {
if (this.use_paging && !this.isFederated()) { //TODO: paging should be refactored using queueing
if (this.page >= (this.page_count)) {
logger.info('All pages processed!');
this.db.close();
this.onDone(this);
} else {
const context = this.getCurrentContext()
if (context.page) {
context.page++
this.page++;
} else {
context.page = ++this.page;
}
logger.debug(`Switching page to ${this.page}`);
let exitCallback = this.onDone;
this.getSourceData(context)
.then(this.processItems.bind(this))
.catch((err) => {
logger.error(err);
exitCallback()
});
}
}
}
}
}
index++;
}).catch((reason) => {
logger.error(reason);
return this.onDone(this);
});
})
}
}
module.exports = AbstractAdapter;
================================================
FILE: src/adapters/factory.js
================================================
'use strict';
class AdapterFactory {
constructor (app_config) {
this.config = app_config;
}
getAdapter (adapter_type, driver) {
let adapter_class = require('./' + adapter_type + '/' + driver);
if (!adapter_class) {
throw new Error(`Invalid adapter ${adapter_type} / ${driver}`);
} else {
let adapter_instance = new adapter_class(this.config);
if((typeof adapter_instance.isValidFor == 'function') && !adapter_instance.isValidFor(driver))
throw new Error(`Not valid adapter class or adapter is not valid for ${driver}`);
return adapter_instance;
}
}
}
module.exports = AdapterFactory;
================================================
FILE: src/adapters/magento/abstract.js
================================================
'use strict';
let AbstractAdapter = require('../abstract');
class AbstractMagentoAdapter extends AbstractAdapter{
constructor(config){
super(config);
let Magento2Client = require('./magento2-rest-client').Magento2Client;
this.api = Magento2Client(this.config.magento);
}
getEntityType(){
throw new Error('getEntityType must be implemented');
}
getCollectionName(){
return this.getEntityType();
}
validateConfig(config){
super.validateConfig(config);
if(!config['magento']['url'] ||
!config['magento']['consumerKey'] ||
!config['magento']['consumerSecret'] ||
!config['magento']['accessToken'] ||
!config['magento']['accessTokenSecret'])
throw Error('magento.{url,consumerKey,consumerSecret,accessToken,accessTokenSecret} must be set in config');
}
isValidFor(entity_type){
return (entity_type == this.getEntityType());
}
getSourceData(){
throw new Error('getSourceData must be implemented');
}
getLabel(source_item){
return source_item.id;
}
/**
* We're transorming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
return item;
}
}
module.exports = AbstractMagentoAdapter;
================================================
FILE: src/adapters/magento/attribute.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
const CacheKeys = require('./cache_keys');
const util = require('util');
class AttributeAdapter extends AbstractMagentoAdapter {
getEntityType() {
return 'attribute';
}
getName() {
return 'adapters/magento/AttributeAdapter';
}
getSourceData(context) {
return this.api.attributes.list();
}
/** Regarding Magento2 api docs and reality we do have an exception here that items aren't listed straight in the response but under "items" key */
prepareItems(items) {
if(!items)
return items;
if (items.total_count)
this.total_count = items.total_count;
if(items.items)
items = items.items; // this is an exceptional behavior for Magento2 api for attributes
return items;
}
getLabel(source_item) {
return `[(${source_item.attribute_code}) ${source_item.default_frontend_label}]`;
}
isFederated() {
return false;
}
preProcessItem(item) {
return new Promise((done, reject) => {
if (item) {
item.id = item.attribute_id;
// store the item into local redis cache
let key = util.format(CacheKeys.CACHE_KEY_ATTRIBUTE, item.attribute_code);
logger.debug(`Storing attribute data to cache under: ${key}`);
this.cache.set(key, JSON.stringify(item));
key = util.format(CacheKeys.CACHE_KEY_ATTRIBUTE, item.attribute_id); // store under attribute id as an second option
logger.debug(`Storing attribute data to cache under: ${key}`);
this.cache.set(key, JSON.stringify(item));
}
return done(item);
});
}
/**
* We're transorming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
return item;
}
}
module.exports = AttributeAdapter;
================================================
FILE: src/adapters/magento/cache_keys.js
================================================
var config = require('../../config')
module.exports = {
CACHE_KEY_CATEGORY: config.db.indexName + '_cat_%s',
CACHE_KEY_PRODUCT: config.db.indexName + '_prd_%s',
CACHE_KEY_PRODUCT_CATEGORIES: config.db.indexName + '_prd_cat_%s',
CACHE_KEY_PRODUCT_CATEGORIES_TEMPORARY: config.db.indexName + '_prd_cat_ts_%s',
CACHE_KEY_ATTRIBUTE: config.db.indexName + '_attr_%s',
CACHE_KEY_STOCKITEM: config.db.indexName + '_stock_%s'
}
================================================
FILE: src/adapters/magento/category.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
const CacheKeys = require('./cache_keys');
const util = require('util');
const request = require('request');
const _slugify = require('../../helpers/slugify')
const _normalizeExtendedData = function (result, generateUrlKey = true, config = null) {
if (result.custom_attributes) {
for (let customAttribute of result.custom_attributes) { // map custom attributes directly to document root scope
result[customAttribute.attribute_code] = customAttribute.value;
}
delete result.custom_attributes;
}
if (generateUrlKey) {
result.url_key = _slugify(result.name) + '-' + result.id;
}
result.slug = result.url_key
if (config.seo.useUrlDispatcher) {
result.url_path = config.seo.categoryUrlPathMapper(result)
} else {
result.url_path = result.url_key;
}
return result
}
class CategoryAdapter extends AbstractMagentoAdapter {
constructor (config) {
super(config);
this.extendedCategories = false;
this.generateUniqueUrlKeys = true;
}
getEntityType() {
return 'category';
}
getName() {
return 'adapters/magento/CategoryAdapter';
}
getSourceData(context) {
this.generateUniqueUrlKeys = context.generateUniqueUrlKeys;
this.extendedCategories = context.extendedCategories;
return this.api.categories.list();
}
getLabel(source_item) {
return `[(${source_item.id}) ${source_item.name}]`;
}
isFederated() {
return false;
}
_addSingleCategoryData(item, result) {
item = Object.assign(item, _normalizeExtendedData(result, this.generateUniqueUrlKeys, this.config));
}
_extendSingleCategory(rootId, catToExtend) {
const generateUniqueUrlKeys = this.generateUniqueUrlKeys
const config = this.config
return this.api.categories.getSingle(catToExtend.id).then(function(result) {
Object.assign(catToExtend, _normalizeExtendedData(result, generateUniqueUrlKeys, config))
logger.info(`Subcategory data extended for ${rootId}, children object ${catToExtend.id}`)
}).catch(function(err) {
logger.error(err)
});
}
_extendChildrenCategories(rootId, children, result, subpromises) {
for (const child of children) {
if (Array.isArray(child.children_data)) {
this._extendChildrenCategories(rootId, child.children_data, result, subpromises);
subpromises.push(this._extendSingleCategory(rootId, child));
} else {
subpromises.push(this._extendSingleCategory(rootId, child));
}
}
return result;
};
preProcessItem(item) {
return new Promise((done, reject) => {
if (!item) {
return done(item);
}
if (!item.url_key || this.generateUniqueUrlKeys) {
item.url_key = _slugify(item.name) + '-' + item.id
}
item.slug = item.url_key;
if (this.config.seo.useUrlDispatcher) {
item.url_path = this.config.seo.categoryUrlPathMapper(item)
} else {
item.url_path = item.url_key;
}
if (this.extendedCategories) {
this.api.categories.getSingle(item.id).then((result) => {
this._addSingleCategoryData(item, result);
const key = util.format(CacheKeys.CACHE_KEY_CATEGORY, item.id);
logger.debug(`Storing extended category data to cache under: ${key}`);
this.cache.set(key, JSON.stringify(item));
const subpromises = []
if (item.children_data && item.children_data.length) {
this._extendChildrenCategories(item.id, item.children_data, result, subpromises)
Promise.all(subpromises).then(function (results) {
done(item)
}).catch(function (err) {
logger.error(err)
done(item)
})
} else {
done(item);
}
}).catch(function (err) {
logger.error(err);
done(item);
});
} else {
const key = util.format(CacheKeys.CACHE_KEY_CATEGORY, item.id);
logger.debug(`Storing category data to cache under: ${key}`);
this.cache.set(key, JSON.stringify(item));
return done(item);
}
});
}
/**
* We're transforming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
if (this.config.vuestorefront && this.config.vuestorefront.invalidateCache) {
request(this.config.vuestorefront.invalidateCacheUrl + 'C' + item.id, {}, (err, res, body) => {
if (err) { return console.error(err); }
console.log(body);
});
}
return item;
}
}
module.exports = CategoryAdapter;
================================================
FILE: src/adapters/magento/cms_block.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
class BlockAdapter extends AbstractMagentoAdapter {
constructor(config) {
super(config);
this.use_paging = false;
}
getEntityType() {
return 'cms_block';
}
getName() {
return 'adapters/magento/BlockAdapter';
}
getSourceData(context) {
if (this.use_paging) {
return this.api.blocks.list('&searchCriteria[currentPage]=' + this.page + '&searchCriteria[pageSize]=' + this.page_size + (query ? '&' + query : '')).catch((err) => {
throw new Error(err);
});
}
return this.api.blocks.list().catch((err) => {
throw new Error(err);
});
}
prepareItems(items) {
if(!items) {
return items;
}
if (items.total_count) {
this.total_count = items.total_count;
}
if (items.items) {
items = items.items; // this is an exceptional behavior for Magento2 api for lists
}
return items;
}
isFederated() {
return false;
}
preProcessItem(item) {
//
return new Promise((done, reject) => {
if (item) {
item.type = 'cms_block'
}
return done(item);
});
}
/**
* We're transorming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
return item;
}
}
module.exports = BlockAdapter;
================================================
FILE: src/adapters/magento/cms_page.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
class PageAdapter extends AbstractMagentoAdapter {
constructor(config) {
super(config);
this.use_paging = false;
}
getEntityType() {
return 'cms_page';
}
getName() {
return 'adapters/magento/PageAdapter';
}
getSourceData(context) {
if (this.use_paging) {
return this.api.pages.list('&searchCriteria[currentPage]=' + this.page + '&searchCriteria[pageSize]=' + this.page_size + (query ? '&' + query : '')).catch((err) => {
throw new Error(err);
});
}
return this.api.pages.list().catch((err) => {
throw new Error(err);
});
}
prepareItems(items) {
if(!items)
return items;
if (items.total_count)
this.total_count = items.total_count;
if (items.items) {
items = items.items; // this is an exceptional behavior for Magento2 api for lists
}
return items;
}
preProcessItem(item) {
return new Promise((done, reject) => {
if (item) {
item.type = 'cms_page'
}
return done(item);
});
}
normalizeDocumentFormat(item) {
return item;
}
}
module.exports = PageAdapter;
================================================
FILE: src/adapters/magento/magento2-rest-client/.npmignore
================================================
# Logs
logs
*.log
npm-debug.log*
# Runtime data
pids
*.pid
*.seed
# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov
# Coverage directory used by tools like istanbul
coverage
# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt
# node-waf configuration
.lock-wscript
# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release
# Dependency directory
node_modules
# Optional npm cache directory
.npm
# Optional REPL history
.node_repl_history
================================================
FILE: src/adapters/magento/magento2-rest-client/LICENSE
================================================
The MIT License (MIT)
Copyright (c) 2016 Marko Novak
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: src/adapters/magento/magento2-rest-client/README.md
================================================
# Magento2 REST client
This Node.js library enables JavaScript applications to communicate with Magento2 sites using their REST API.
**NOTE: the library is not finished yet! Only a subset of Magento2 API is currently implemented.**
## Installation
The library can be installed using the Npm package manager:
```
npm install magento2-rest-client
```
## Usage
The code sample below shows the usage of the library:
```javascript
var Magento2Client = require('magento2-rest-client').Magento2Client;
var options = {
'url': 'http://www.test.com/index.php/rest',
'consumerKey': '<OAuth 1.0a consumer key>',
'consumerSecret': '<OAuth 1.0a consumer secret>',
'accessToken': '<OAuth 1.0a access token>',
'accessTokenSecret': '<OAuth 1.0a access token secret>'
};
var client = Magento2Client(options);
client.categories.list()
.then(function (categories) {
assert.equal(categories.parentId, 1);
})
```
================================================
FILE: src/adapters/magento/magento2-rest-client/index.js
================================================
'use strict';
var RestClient = require('./lib/rest_client').RestClient;
var categories = require('./lib/categories');
var attributes = require('./lib/attributes');
var products = require('./lib/products');
var productMedia = require('./lib/product_media');
var categoryProducts = require('./lib/category_products');
var configurableChildren = require('./lib/configurable_children');
var configurableOptions = require('./lib/configurable_options');
var customOptions = require('./lib/custom_options');
var bundleOptions = require('./lib/bundle_options');
var taxRates = require('./lib/tax_rates');
var taxRules = require('./lib/tax_rules');
var stockItems = require('./lib/stock_items');
var productLinks = require('./lib/product_links');
var reviews = require('./lib/reviews');
var blocks = require('./lib/blocks');
var pages = require('./lib/pages');
const MAGENTO_API_VERSION = 'V1';
module.exports.Magento2Client = function (options) {
var instance = {};
options.version = MAGENTO_API_VERSION;
var client = RestClient(options);
instance.attributes = attributes(client);
instance.categories = categories(client);
instance.products = products(client);
instance.productMedia = productMedia(client);
instance.categoryProducts = categoryProducts(client);
instance.configurableChildren = configurableChildren(client);
instance.configurableOptions = configurableOptions(client);
instance.stockItems = stockItems(client);
instance.taxRates = taxRates(client);
instance.taxRules = taxRules(client);
instance.customOptions = customOptions(client);
instance.bundleOptions = bundleOptions(client);
instance.productLinks = productLinks(client);
instance.reviews = reviews(client);
instance.blocks = blocks(client);
instance.pages = pages(client);
return instance;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/attributes.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (searchCriteria) {
var query = 'searchCriteria=' + searchCriteria;
var endpointUrl = util.format('/products/attributes?%s', query);
return restClient.get(endpointUrl);
}
module.create = function (categoryAttributes) {
return restClient.post('/products/attributes', categoryAttributes);
}
module.update = function (attributeId, categoryAttributes) {
var endpointUrl = util.format('/products/attributes/%d', attributeId);
return restClient.put(endpointUrl, categoryAttributes);
}
module.delete = function (attributeId) {
var endpointUrl = util.format('/products/attributes/%d', attributeId);
return restClient.delete(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/blocks.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function(searchCriteria) {
var query = 'searchCriteria=' + searchCriteria;
var endpointUrl = util.format('/snowdog/cmsBlock/search?%s', query);
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/bundle_options.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (sku) {
var endpointUrl = util.format('/bundle-products/%s/options/all', encodeURIComponent(sku));
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/categories.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function () {
return restClient.get('/categories');
}
module.getSingle = function (categoryId) {
var endpointUrl = util.format('/categories/%d', categoryId);
return restClient.get(endpointUrl);
}
module.create = function (categoryAttributes) {
return restClient.post('/categories', categoryAttributes);
}
module.update = function (categoryId, categoryAttributes) {
var endpointUrl = util.format('/categories/%d', categoryId);
return restClient.put(endpointUrl, categoryAttributes);
}
module.delete = function (categoryId) {
var endpointUrl = util.format('/categories/%d', categoryId);
return restClient.delete(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/category_products.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (categoryId) {
var endpointUrl = util.format('/categories/%d/products', categoryId);
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/configurable_children.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (sku) {
var endpointUrl = util.format('/configurable-products/%s/children', encodeURIComponent(sku));
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/configurable_options.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (sku) {
var endpointUrl = util.format('/configurable-products/%s/options/all', encodeURIComponent(sku));
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/custom_options.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (sku) {
var endpointUrl = util.format('/products/%s/options', encodeURIComponent(sku));
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/log.js
================================================
var winston = require('winston');
winston.emitErrs = true;
var logger = new winston.Logger({
transports: [
new winston.transports.Console({
level: 'debug',
handleExceptions: true,
json: false,
colorize: true
})
],
exitOnError: false
});
logger.info('Winston logging library initialized.');
module.exports = logger;
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/pages.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (searchCriteria) {
var query = 'searchCriteria=' + searchCriteria;
var endpointUrl = util.format('/snowdog/cmsPage/search?%s', query);
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/product_links.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
var typesCache = null;
module.list = function (sku, type) {
var endpointUrl = util.format('/products/%s/links/%s', encodeURIComponent(sku), type);
return restClient.get(endpointUrl);
}
module.types = function () {
var endpointUrl = util.format('/products/links/types');
if (typesCache !== null) {
return new Promise((resolve, reject) => {
resolve (typesCache)
})
} else {
return restClient.get(endpointUrl).then((result) => {
typesCache = result
})
}
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/product_media.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (productSku) {
var endpointUrl = util.format('/products/%s/media', encodeURIComponent(productSku));
return restClient.get(endpointUrl);
}
module.get = function (productSku, mediaId) {
var endpointUrl = util.format('/products/%s/media/%d', encodeURIComponent(productSku), mediaId);
return restClient.get(endpointUrl);
}
module.create = function (productSku, productMediaAttributes) {
var endpointUrl = util.format('/products/%s/media', encodeURIComponent(productSku));
return restClient.post(endpointUrl, productMediaAttributes);
}
module.update = function (productSku, mediaId, productMediaAttributes) {
var endpointUrl = util.format('/products/%s/media/%d', encodeURIComponent(productSku), mediaId);
return restClient.put(endpointUrl, productMediaAttributes);
}
module.delete = function (productSku, mediaId) {
var endpointUrl = util.format('/products/%s/media/%d', encodeURIComponent(productSku), mediaId);
return restClient.delete(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/products.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (searchCriteria) {
var query = 'searchCriteria=' + searchCriteria;
var endpointUrl = util.format('/products?%s', query);
return restClient.get(endpointUrl);
}
module.renderList = function (searchCriteria, storeId = 1, currencyCode = 'USD') {
var query = 'searchCriteria=' + searchCriteria;
var endpointUrl = util.format('/products-render-info?%s&storeId=%d¤cyCode=' + encodeURIComponent(currencyCode), query, storeId);
return restClient.get(endpointUrl);
}
module.create = function (productAttributes) {
return restClient.post('/products', productAttributes);
}
module.update = function (productSku, productAttributes) {
var endpointUrl = util.format('/products/%s', encodeURIComponent(productSku));
return restClient.put(endpointUrl, productAttributes);
}
module.delete = function (productSku) {
var endpointUrl = util.format('/products/%s', encodeURIComponent(productSku));
return restClient.delete(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/rest_client.js
================================================
'use strict';
var OAuth = require('oauth-1.0a');
var request = require('request');
var humps = require('humps');
var sprintf = require('util').format;
var logger = require('./log');
module.exports.RestClient = function (options) {
var instance = {};
var servelrUrl = options.url;
var apiVersion = options.version;
var oauth = OAuth({
consumer: {
public: options.consumerKey,
secret: options.consumerSecret
},
signature_method: 'HMAC-SHA1'
});
var token = {
public: options.accessToken,
secret: options.accessTokenSecret
};
function apiCall(request_data) {
logger.debug('Calling API endpoint: ' + request_data.method + ' ' + request_data.url);
return new Promise(function (resolve, reject) {
request({
url: request_data.url,
method: request_data.method,
headers: oauth.toHeader(oauth.authorize(request_data, token)),
json: true,
body: request_data.body,
}, function (error, response, body) {
logger.debug('Response received.');
if (error) {
logger.error('Error occured: ' + error);
reject(error);
return;
} else if (!httpCallSucceeded(response)) {
var errorMessage = 'HTTP ERROR ' + response.code;
if(body && body.hasOwnProperty('message') )
errorMessage = errorString(body.message, body.hasOwnProperty('parameters') ? body.parameters : {});
logger.error('API call failed: ' + errorMessage);
reject(errorMessage);
}
// var bodyCamelized = humps.camelizeKeys(body);
// resolve(bodyCamelized);
resolve(body);
});
});
}
function httpCallSucceeded(response) {
return response.statusCode >= 200 && response.statusCode < 300;
}
function errorString(message, parameters) {
if (parameters === null) {
return message;
}
if (parameters instanceof Array) {
for (var i = 0; i < parameters.length; i++) {
var parameterPlaceholder = '%' + (i + 1).toString();
message = message.replace(parameterPlaceholder, parameters[i]);
}
} else if (parameters instanceof Object) {
for (var key in parameters) {
var parameterPlaceholder = '%' + key;
message = message.replace(parameterPlaceholder, parameters[key]);
}
}
return message;
}
instance.get = function (resourceUrl) {
var request_data = {
url: createUrl(resourceUrl),
method: 'GET'
};
return apiCall(request_data);
}
function createUrl(resourceUrl) {
return servelrUrl + '/' + apiVersion + resourceUrl;
}
instance.post = function (resourceUrl, data) {
var request_data = {
url: createUrl(resourceUrl),
method: 'POST',
body: data
};
return apiCall(request_data);
}
instance.put = function (resourceUrl, data) {
var request_data = {
url: createUrl(resourceUrl),
method: 'PUT',
body: data
};
return apiCall(request_data);
}
instance.delete = function (resourceUrl) {
var request_data = {
url: createUrl(resourceUrl),
method: 'DELETE'
};
return apiCall(request_data);
}
return instance;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/reviews.js
================================================
const util = require('util');
module.exports = function (restClient) {
var module = {};
module.getByProductSku = function (sku) {
const endpointUrl = util.format('/products/%s/review', encodeURIComponent(sku));
return restClient.get(endpointUrl);
};
module.list = function(searchCriteria) {
const query = 'searchCriteria=' + searchCriteria;
const endpointUrl = util.format('/reviews/?%s', query);
return restClient.get(endpointUrl);
};
module.create = function (reviewData) {
return restClient.post('/reviews', {review: reviewData})
}
module.delete = function (reviewId) {
var endpointUrl = util.format('/reviews/%d', reviewId);
return restClient.delete(endpointUrl);
}
return module;
};
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/stock_items.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (sku) {
var endpointUrl = util.format('/stockItems/%s', encodeURIComponent(sku));
return restClient.get(endpointUrl);
}
// MSI
module.getSalableQty = function (sku, stockId) {
var endpointUrl = util.format(
'/inventory/get-product-salable-quantity/%s/%d',
encodeURIComponent(sku),
encodeURIComponent(stockId)
);
return restClient.get(endpointUrl);
}
// MSI
module.isSalable = function (sku, stockId) {
var endpointUrl = util.format(
'/inventory/is-product-salable/%s/%d',
encodeURIComponent(sku),
encodeURIComponent(stockId)
);
return restClient.get(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/tax_rates.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (rateId) {
var endpointUrl = util.format('/taxRates/%d', rateId);
return restClient.get(endpointUrl);
}
module.create = function (rateAttributes) {
return restClient.post('/taxRates', rateAttributes);
}
module.update = function (rateId, rateAttributes) {
var endpointUrl = util.format('/taxRates/%d', rateId);
return restClient.put(endpointUrl, rateAttributes);
}
module.delete = function (rateId) {
var endpointUrl = util.format('/taxRates/%d', rateId);
return restClient.delete(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/lib/tax_rules.js
================================================
var util = require('util');
module.exports = function (restClient) {
var module = {};
module.list = function (searchCriteria) {
var query = 'searchCriteria=' + searchCriteria;
var endpointUrl = util.format('/taxRules/search?%s', query);
return restClient.get(endpointUrl);
}
module.create = function (ruleAttributes) {
return restClient.post('/taxRules', ruleAttributes);
}
module.update = function (ruleId, ruleAttributes) {
var endpointUrl = util.format('/taxRules/%d', ruleId);
return restClient.put(endpointUrl, ruleAttributes);
}
module.delete = function (ruleId) {
var endpointUrl = util.format('/taxRules/%d', ruleId);
return restClient.delete(endpointUrl);
}
return module;
}
================================================
FILE: src/adapters/magento/magento2-rest-client/magento2-rest-client.iml
================================================
<?xml version="1.0" encoding="UTF-8"?>
<module type="WEB_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$" />
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="magento2-rest-client node_modules" level="project" />
</component>
</module>
================================================
FILE: src/adapters/magento/magento2-rest-client/package.json
================================================
{
"_from": "magento2-rest-client@0.0.2",
"_id": "magento2-rest-client@0.0.2",
"_inBundle": false,
"_integrity": "sha1-Km7yMMiBKahoC2Pq03k+p+mb9qQ=",
"_location": "/magento2-rest-client",
"_phantomChildren": {},
"_requested": {
"type": "version",
"registry": true,
"raw": "magento2-rest-client@0.0.2",
"name": "magento2-rest-client",
"escapedName": "magento2-rest-client",
"rawSpec": "0.0.2",
"saveSpec": null,
"fetchSpec": "0.0.2"
},
"_requiredBy": [
"/"
],
"_resolved": "https://registry.npmjs.org/magento2-rest-client/-/magento2-rest-client-0.0.2.tgz",
"_shasum": "2a6ef230c88129a8680b63ead3793ea7e99bf6a4",
"_spec": "magento2-rest-client@0.0.2",
"_where": "/Users/pkarwatka/Documents/_PROJEKTY/mage2nosql/src",
"author": {
"name": "Marko Novak",
"email": "nouvak@gmail.com"
},
"bugs": {
"url": "https://github.com/nouvak/magento2-rest-client/issues"
},
"bundleDependencies": false,
"dependencies": {
"humps": "^1.1.0",
"oauth-1.0a": "^1.0.1",
"request": "^2.72.0",
"winston": "^2.2.0"
},
"deprecated": false,
"description": "REST client for accessing Magento 2 functionality.",
"devDependencies": {
"chai": "^3.5.0",
"mocha": "^2.4.5"
},
"directories": {
"lib": "./lib"
},
"homepage": "https://github.com/nouvak/magento2-rest-client#readme",
"keywords": [
"magento2",
"REST",
"API"
],
"license": "MIT",
"main": "index.js",
"name": "magento2-rest-client",
"repository": {
"type": "git",
"url": "git+https://github.com/nouvak/magento2-rest-client.git"
},
"scripts": {
"test": "test"
},
"version": "0.0.2"
}
================================================
FILE: src/adapters/magento/magento2-rest-client/test/config.json
================================================
{
"url": "http://www.igracke.si/index.php/rest",
"consumerKey": "h0vun4y13ov0pg1drcndfhnim761omcu",
"consumerSecret": "prdemsvf7usypg1f6q24cpis2k1pdtkw",
"accessToken": "kdrmfbudawyquoelylm6syjtb94hmalw",
"accessTokenSecret": "gqjls6qef730m3mniid443m799cu1i8d"
}
================================================
FILE: src/adapters/magento/magento2-rest-client/test/integration/categories.integration.test.js
================================================
var chai = require('chai');
var credentials = require('../config');
var assert = chai.assert;
var Magento2Client = require('../../index').Magento2Client;
suite('categories tests', function () {
var client;
before(function() {
client = Magento2Client(credentials);
});
test('list categories test', function (done) {
client.categories.list()
.then(function (categories) {
assert.equal(categories.parentId, 1);
})
.then(done, done);
});
test('create category test', function (done) {
var newCategory = {
category: {
parentId: 3,
name: 'Category from integration test',
isActive: true,
includeInMenu: true,
}
};
client.categories.create(newCategory)
.then(function (result) {
assert.equal(result.parentId, 3);
})
.then(done, done);
});
test('update category test', function (done) {
var categoryUpdate = {
category: {
parentId: 3,
name: 'Podkategorija 1 updated',
isActive: true,
includeInMenu: true,
}
};
client.categories.update(4, categoryUpdate)
.then(function (result) {
assert.equal(result.parentId, 3);
})
.then(done, done);
});
test('delete category test', function (done) {
client.categories.delete(23)
.then(function (result) {
assert.isTrue(result);
})
.then(done, done);
})
});
================================================
FILE: src/adapters/magento/magento2-rest-client/test/integration/product_media.integration.test.js
================================================
var chai = require('chai');
var credentials = require('../config');
var assert = chai.assert;
var Magento2Client = require('../../index').Magento2Client;
suite('products media tests', function () {
var client;
before(function () {
client = Magento2Client(credentials);
});
test('list product media test', function (done) {
client.productMedia.list('test123')
.then(function (productMedia) {
assert.isTrue(productMedia.length > 0);
})
.then(done, done);
});
test('get product media test', function (done) {
client.productMedia.get('test123', 15)
.then(function (productMedia) {
assert.isNotNull(productMedia);
})
.then(done, done);
});
test('create product media test', function (done) {
var newProductMedia = {
'entry': {
'media_type': 'image',
'label': 'Image',
'position': 1,
'disabled': false,
'types': [
'image',
'small_image',
'thumbnail'
],
'file': '/m/b/mb01-blue-0.png',
'content': {
'base64EncodedData': 'iVBORw0KGgoAAAANSUhEUgAAACgAAAAoCAYAAACM/rhtAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAWtJREFUeNpi/P//P8NgBkwMgxyMOnDUgTDAyMhIDNYF4vNA/B+IDwCxHLoakgEoFxODiQRXQUYi4e3k2gfDjMRajsP3zED8F8pmA+JvUDEYeArEMugOpFcanA/Ef6A0CPwC4uNoag5SnAjJjGI2tKhkg4rLAfFGIH4IxEuBWIjSKKYkDfZCHddLiwChVhokK8YGohwEZYy3aBmEKmDEhOCgreomo+VmZHxsMEQxIc2MAx3FO/DI3RxMmQTZkI9ALDCaSUYdOOrAIeRAPzQ+PxCHUM2FFDb5paGNBPRa5C20bUhxc4sSB4JaLnvxVHWHsbVu6OnACjyOg+HqgXKgGRD/JMKBoD6LDb0dyAPE94hwHAw/hGYcujlwEQmOg+EV9HJgLBmOg+FMWjsQVKR8psCBoDSrQqoDSSmoG6Hpj1wA6ju30LI9+BBX4UsC+Ai0T4BWVd1EIL5PgeO+APECmoXgaGtm1IE0AgABBgAJAICuV8dAUAAAAABJRU5ErkJggg==',
'type': 'image/png',
'name': 'new_image.png'
}
}
};
client.productMedia.create('test123', newProductMedia)
.then(function (result) {
assert.isNotNull(result);
})
.then(done, done);
});
test('update product test', function (done) {
var productMediaUpdate = {
'entry': {
'id': 15,
'label': 'Image updated',
}
};
client.productMedia.update('test123', 15, productMediaUpdate)
.then(function (result) {
assert.isNotNull(result);
})
.then(done, done);
});
test('delete product test', function (done) {
client.productMedia.delete('test123', 10)
.then(function (result) {
assert.isTrue(result);
})
.then(done, done);
})
});
================================================
FILE: src/adapters/magento/magento2-rest-client/test/integration/products.integration.test.js
================================================
var chai = require('chai');
var credentials = require('../config');
var assert = chai.assert;
var Magento2Client = require('../../index').Magento2Client;
suite('products tests', function () {
var client;
before(function() {
client = Magento2Client(credentials);
});
test('list products test', function (done) {
client.products.list('Test')
.then(function (products) {
assert.isTrue(products.totalCount > 0);
})
.then(done, done);
});
test('create product test', function (done) {
var newProduct = {
product: {
'sku': 'test123',
'name': 'Integration test product',
'typeId': 'simple',
'price': 12.3,
'attributeSetId': 4,
'status': 1,
'visibility': 4,
}
};
client.products.create(newProduct)
.then(function (result) {
assert.equal(result.name, 'Integration test product');
})
.then(done, done);
});
test('update product test', function (done) {
var productUpdate = {
product: {
'sku': 'test123',
'name': 'Integration test product updated',
'typeId': 'simple',
'price': 12.3,
'attributeSetId': 4,
'status': 1,
'visibility': 4,
}
};
client.products.update('test123', productUpdate)
.then(function (result) {
assert.equal(result.name, 'Integration test product updated');
})
.then(done, done);
});
test('delete product test', function (done) {
client.products.delete(23)
.then(function (result) {
assert.isTrue(result);
})
.then(done, done);
})
});
================================================
FILE: src/adapters/magento/product.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
const util = require('util');
const CacheKeys = require('./cache_keys');
const moment = require('moment')
const _ = require('lodash')
const request = require('request');
const HTTP_RETRIES = 3
let kue = require('kue');
const _slugify = require('../../helpers/slugify')
/*
* serial executes Promises sequentially.
* @param {funcs} An array of funcs that return promises.
* @example
* const urls = ['/url1', '/url2', '/url3']
* serial(urls.map(url => () => $.ajax(url)))
* .then(console.log(console))
*/
const serial = funcs =>
funcs.reduce((promise, func) =>
promise.then(result => func().then(Array.prototype.concat.bind(result))), Promise.resolve([]))
const optionLabel = (attr, optionId) => {
if (attr) {
let opt = attr.options.find((op) => { // TODO: cache it in memory
if (_.toString(op.value) === _.toString(optionId)) {
return op
}
}) // TODO: i18n support with multi website attribute names
return opt ? opt.label : optionId
} else {
return optionId
}
}
class ProductAdapter extends AbstractMagentoAdapter {
constructor(config) {
super(config);
this.use_paging = true;
this.stock_sync = true;
this.custom_sync = true;
this.parent_sync = true;
this.media_sync = true;
this.category_sync = true;
this.links_sync = true;
this.configurable_sync = true;
this.is_federated = true; // by default use federated behaviour
}
getEntityType() {
return 'product';
}
getName() {
return 'adapters/magento/ProductAdapter';
}
prepareItems(items) {
if(!items)
return null;
this.total_count = items.total_count;
if (this.use_paging) {
this.page_count = Math.ceil(this.total_count / this.page_size);
logger.info('Page count', this.page_count)
}
return items.items;
}
getFilterQuery(context) {
let query = '';
if (context.skus) { // pul individual products
if (!Array.isArray(context.skus))
context.skus = new Array(context.skus);
query += 'searchCriteria[filter_groups][0][filters][0][field]=sku&' +
'searchCriteria[filter_groups][0][filters][0][value]=' + encodeURIComponent(context.skus.join(',')) + '&' +
'searchCriteria[filter_groups][0][filters][0][condition_type]=in';
} else if (context.updated_after && typeof context.updated_after == 'object') {
query += 'searchCriteria[filter_groups][0][filters][0][field]=updated_at&' +
'searchCriteria[filter_groups][0][filters][0][value]=' + encodeURIComponent(moment(context.updated_after).utc().format()) + '&' +
'searchCriteria[filter_groups][0][filters][0][condition_type]=gt';
}
return query;
}
getSourceData(context) {
const that = this
const retryHandler = (context, err, reject) => {
context.retry_count = context.retry_count ? context.retry_count + 1 : 1;
if (err == null || context.retry_count < HTTP_RETRIES ) {
if (err) {
logger.error(err);
logger.info('Retrying getSourceData() request ' + context.retry_count);
}
if (this.config.product && this.config.product.synchronizeCatalogSpecialPrices) {
return new Promise((resolve, reject) => {
this.getProductSourceData(context).then((result) => {
// download rendered list items
const products = result.items
let skus = products.map((p) => { return p.sku })
if (products.length === 1) { // single product - download child data
const childSkus = _.flattenDeep(products.map((p) => { return (p.configurable_children) ? p.configurable_children.map((cc) => { return cc.sku }) : null }))
skus = _.union(skus, childSkus)
}
const query = '&searchCriteria[filter_groups][0][filters][0][field]=sku&' +
'searchCriteria[filter_groups][0][filters][0][value]=' + encodeURIComponent(skus.join(',')) + '&' +
'searchCriteria[filter_groups][0][filters][0][condition_type]=in';
this.api.products.renderList(query, this.config.magento.storeId, this.config.magento.currencyCode).then(renderedProducts => {
context.renderedProducts = renderedProducts
for (let product of result.items) {
const productAdditionalInfo = renderedProducts.items.find(p => p.id === product.id)
if (productAdditionalInfo && productAdditionalInfo.price_info) {
delete productAdditionalInfo.price_info.formatted_prices
delete productAdditionalInfo.price_info.extension_attributes
// delete productAdditionalInfo.price_info.special_price
product = Object.assign(product, productAdditionalInfo.price_info)
if (product.final_price < product.price) {
product.special_price = product.final_price
}
if (this.config.product.renderCatalogRegularPrices) {
product.price = product.regular_price
}
}
}
resolve(result)
})
}).catch(err => {
retryHandler(context, err, reject)
})
})
} else {
return this.getProductSourceData(context).catch(err => {
retryHandler(context, err, null)
})
}
} else {
if (reject) {
reject(err)
} else {
throw err
}
}
}
// run the import logick
return retryHandler(context, null, null)
}
getProductSourceData(context) {
let query = this.getFilterQuery(context);
let searchCriteria = '&searchCriteria[currentPage]=%d&searchCriteria[pageSize]=%d';
if(this.config.product && JSON.parse(this.config.product.excludeDisabledProducts)) {
searchCriteria += '&searchCriteria[filterGroups][0][filters][0][field]=status'+
'&searchCriteria[filterGroups][0][filters][0][value]=1';
}
if(typeof context.stock_sync !== 'undefined')
this.stock_sync = context.stock_sync;
if(typeof context.parent_sync !== 'undefined')
{
logger.info('Configurable parent sync is ', context.parent_sync)
this.parent_sync = context.parent_sync;
}
if(typeof context.category_sync !== 'undefined')
this.category_sync = context.category_sync;
if(typeof context.configurable_sync !== 'undefined')
this.configurable_sync = context.configurable_sync;
if (context.for_total_count) { // get total counts
return this.api.products.list(util.format(searchCriteria, 1, 1)).catch((err) => {
throw new Error(err);
});
} else if (context.page && context.page_size) {
this.use_paging = context.use_paging || false
this.is_federated = context.use_paging ? false : true;
this.page = context.page;
this.page_size = context.page_size
if (!context.use_paging) this.page_count = 1; // process only one page - used for partitioning purposes
logger.debug(`Using specific paging options from adapter context: ${context.page} / ${context.page_size}`);
return this.api.products.list(util.format(searchCriteria, context.page, context.page_size) + (query ? '&' + query : '')).catch((err) => {
throw new Error(err);
});
} else if (this.use_paging) {
this.is_federated = false; // federated execution is not compliant with paging
logger.debug(util.format(searchCriteria, this.page, this.page_size) + (query ? '&' + query : ''));
return this.api.products.list(util.format(searchCriteria, this.page, this.page_size) + (query ? '&' + query : '')).catch((err) => {
throw new Error(err);
});
} else {
return this.api.products.list().catch((err) => {
throw new Error(err);
});
}
}
getTotalCount(context) {
context = context ? Object.assign(context, { for_total_count: 1 }) : { for_total_count: 1 };
return this.getSourceData(context); //api.products.list('&searchCriteria[currentPage]=1&searchCriteria[pageSize]=1');
}
getLabel(source_item) {
return `[(${source_item.id} - ${source_item.sku}) ${source_item.name}]`;
}
isNumeric(value) {
return /^\d+$/.test(value);
}
processAttributes(customAttributes, configurableOptions) {
const loadFromCache = (key) => new Promise((resolve) =>
this.cache.get(key, (err, serializedAtr) => resolve(JSON.parse(serializedAtr)))
)
const findConfigurableOptionsValues = attributeId => {
const attribute = configurableOptions.find(
opt => parseInt(opt.attribute_id) === parseInt(attributeId)
)
if (attribute) {
return attribute.values.map(val => parseInt(val.value_index))
}
return []
}
const findCustomAttributesValues = (attributeCode) => {
const attribute = customAttributes.find(
opt => opt.attribute_code === attributeCode
)
return attribute ? [parseInt(attribute.value)] : []
}
const findOptionValues = option => ([
...findConfigurableOptionsValues(option.attribute_id),
...findCustomAttributesValues(option.attribute_code)
])
const selectFields = (res) => res.map(o => {
const attributeOptionValues = findOptionValues(o)
const options = o.options.filter(opt => attributeOptionValues.includes(parseInt(opt.value)))
return {
is_visible_on_front: o.is_visible_on_front,
is_visible: o.is_visible,
default_frontend_label: o.default_frontend_label,
attribute_id: o.attribute_id,
entity_type_id: o.entity_type_id,
id: o.id,
frontend_input: o.frontend_input,
is_user_defined: o.is_user_defined,
is_comparable: o.is_comparable,
attribute_code: o.attribute_code,
slug: o.slug,
options
}
})
const attributeCodes = customAttributes.map(obj => new Promise((resolve) => {
const key = util.format(CacheKeys.CACHE_KEY_ATTRIBUTE, obj.attribute_code);
loadFromCache(key).then(resolve)
}))
const attributeIds = configurableOptions.map(obj => new Promise((resolve) => {
const key = util.format(CacheKeys.CACHE_KEY_ATTRIBUTE, obj.attribute_id);
loadFromCache(key).then(resolve)
}))
return Promise.all([
...attributeCodes,
...attributeIds
])
.then(selectFields)
}
/**
*
* @param {Object} item
*/
preProcessItem(item) {
for (let customAttribute of item.custom_attributes) { // map custom attributes directly to document root scope
let valueArray = String(customAttribute['value']).split(',');
let attrValue = valueArray.map(Number);
if (valueArray.length > 1){
for (let element of valueArray){
if (!this.isNumeric(element)) {
attrValue = customAttribute.value;
break;
}
}
} else {
attrValue = customAttribute.value;
}
item[customAttribute.attribute_code] = attrValue;
}
item.slug = _slugify(item.name + '-' + item.id);
return new Promise((done, reject) => {
// TODO: add denormalization of productcategories into product categories
// DO NOT use "productcategories" type but rather do search categories with assigned products
let subSyncPromises = [];
const config = this.config;
// TODO: Refactor the following to "Chain of responsibility"
// STOCK SYNC
if (this.stock_sync) {
logger.info(`Product sub-stage 1: Getting stock items for ${item.sku}`);
subSyncPromises.push(() => {
return this.api.stockItems.list(item.sku).then((result) => {
item.stock = result;
if (this.config.magento.msi.enabled) {
return this.api.stockItems.getSalableQty(item.sku, this.config.magento.msi.stockId).then((salableQty) => {
item.stock.qty = salableQty;
return item;
}).then((item) => {
return this.api.stockItems.isSalable(item.sku, this.config.magento.msi.stockId).then((isSalable) => {
item.stock.is_in_stock = isSalable;
const key = util.format(CacheKeys.CACHE_KEY_STOCKITEM, item.id);
logger.debug(`Storing stock data to cache under: ${key}`);
this.cache.set(key, JSON.stringify(item.stock));
return item;
})
})
} else {
const key = util.format(CacheKeys.CACHE_KEY_STOCKITEM, item.id);
logger.debug(`Storing stock data to cache under: ${key}`);
this.cache.set(key, JSON.stringify(result));
return item;
}
})
})
}
// MEDIA SYNC
if (this.media_sync) {
logger.info(`Product sub-stage 2: Getting media gallery ${item.sku}`);
subSyncPromises.push(() => {
return this.api.productMedia.list(item.sku).then((result) => {
let media_gallery = []
for (let mediaItem of result){
if (!mediaItem.disabled) {
media_gallery.push({
image: mediaItem.file,
pos: mediaItem.position,
typ: mediaItem.media_type,
lab: mediaItem.label,
vid: this.computeVideoData(mediaItem)
})
}
}
item.media_gallery = media_gallery
return item
})
})
}
// CUSTOM OPTIONS SYNC
if (this.custom_sync) {
logger.info(`Product sub-stage 3: Getting product custom options ${item.sku}`);
subSyncPromises.push(() => {
return this.api.customOptions.list(item.sku).then((result) => {
if (result && result.length > 0) {
item.custom_options = result
logger.info(`Found custom options for ${item.sku}: ${result.length}`)
}
return item
})
})
}
// BUNDLE OPTIONS SYNC
if (this.custom_sync && item.type_id == 'bundle') {
logger.info(`Product sub-stage 4: Getting bundle custom options ${item.sku}`);
subSyncPromises.push(() => {
return this.api.bundleOptions.list(item.sku).then((result) => {
if(result && result.length > 0) {
item.bundle_options = result
logger.info(`Found bundle options for ${item.sku}: ${result.length}`)
}
return item
})
})
}
// PRODUCT LINKS - as it seems magento returns these links anyway in the "product_links"
if (this.links_sync) {
logger.info(`Product sub-stage 5: Getting product links ${item.sku}`);
item.links = {}
subSyncPromises.push(() => {
return new Promise ((opResolve, opReject) => {
return this.api.productLinks.types().then((result) => {
if(result && result.length > 0) {
let subPromises = []
for (const linkType of result) {
logger.info(`Getting the product links ${item.sku}: ${linkType.name}`)
subPromises.push(this.api.productLinks.list(item.sku, linkType.name).then((links) => {
if(links && links.length > 0) {
item.links[linkType.name] = links.map((r) => { return { sku: r.linked_product_sku, pos: r.position } })
logger.info(`Found related products for ${item.sku}: ${item.links[linkType.name]}`)
}
return item
}))
}
Promise.all(subPromises).then((res) => {
logger.info('Product links expanded!')
opResolve(item)
}).catch((err) => {
logger.error(err)
opResolve(item)
})
} else {
opResolve (item)
}
return item
})
})})
}
if (this.parent_sync && (item.type_id == 'simple')) {
subSyncPromises.push(() => {
return new Promise ((opResolve, opReject) => {
// Find the parent product and schedule a sync after subsequent configurable_children got modified
this.db.getDocuments('product', { query: { match: {'configurable_children.sku': item.sku } }}).then((docs) => {
if (docs && docs.length > 0) {
let queue = kue.createQueue(Object.assign(config.kue, { redis: config.redis }));
docs.map(parentProduct => { // schedule for update
queue.createJob('product', { skus: [parentProduct.sku], adapter: 'magento' }).save();
logger.info('Parent product update scheduled (make sure `cli.js productsworker` queue is running)', parentProduct.sku)
})
opResolve(item)
} else {
opResolve(item)
}
}).catch(err => {
logger.error(err)
opResolve(item)
})
})
})
}
// CONFIGURABLE AND BUNDLE SYNC
if (this.configurable_sync && (item.type_id == 'configurable')) {
logger.info(`Product sub-stage 6: Getting product options for ${item.sku}`);
// q.push(() => {
subSyncPromises.push(() => {
return new Promise ((opResolve, opReject) => {
this.api.configurableChildren.list(item.sku).then((result) => {
item.configurable_children = new Array()
for (let prOption of result) {
let confChild = {
sku: prOption.sku,
id: prOption.id,
status: prOption.status,
visibility: prOption.visibility,
name: prOption.name,
price: prOption.price,
tier_prices: prOption.tier_prices,
};
if (prOption.custom_attributes) {
for (let opt of prOption.custom_attributes) {
confChild[opt.attribute_code] = opt.value
}
}
const context = this.current_context
if (context.renderedProducts && context.renderedProducts.items.length) {
const renderedProducts = context.renderedProducts
const subProductAdditionalInfo = renderedProducts.items.find(p => p.id === confChild.id)
if (subProductAdditionalInfo && subProductAdditionalInfo.price_info) {
delete subProductAdditionalInfo.price_info.formatted_prices
delete subProductAdditionalInfo.price_info.extension_attributes
// delete subProductAdditionalInfo.price_info.special_price // always empty :-(
confChild = Object.assign(confChild, subProductAdditionalInfo.price_info)
if (confChild.final_price < confChild.price) {
confChild.special_price = confChild.final_price
}
if(config.product.renderCatalogRegularPrices) {
confChild.price = confChild.regular_price
}
}
}
item.configurable_children.push(confChild);
if(item.price == 0) // if price is zero fix it with first children
item.price = prOption.price;
}
// EXPAND CONFIGURABLE CHILDREN ATTRS
if (config.product && config.product.expandConfigurableFilters) {
for (const attrToExpand of config.product.expandConfigurableFilters){
const expandedSet = new Set()
if (item[attrToExpand]) {
expandedSet.add(item[attrToExpand])
}
for (const confChild of item.configurable_children) {
if (confChild[attrToExpand]) {
expandedSet.add(confChild[attrToExpand])
}
}
if (expandedSet.size > 0) {
item[attrToExpand + '_options'] = Array.from(expandedSet)
}
}
}
this.api.configurableOptions.list(item.sku).then((result) => {
item.configurable_options = result;
let subPromises = []
for (let option of item.configurable_options) {
let atrKey = util.format(CacheKeys.CACHE_KEY_ATTRIBUTE, option.attribute_id);
subPromises.push(new Promise ((resolve, reject) => {
logger.info(`Configurable options for ${atrKey}`)
this.cache.get(atrKey, (err, serializedAtr) => {
let atr = JSON.parse(serializedAtr); // category object
if (atr != null) {
option.attribute_code = atr.attribute_code;
option.values.map((el) => {
el.label = optionLabel(atr, el.value_index)
} )
logger.info(`Product options for ${atr.attribute_code} for ${item.sku} set`);
item[atr.attribute_code + '_options'] = option.values.map((el) => { return el.value_index } )
}
resolve(item)
})
}))
}
Promise.all(subPromises).then((res) => {
logger.info('Configurable options expanded!')
opResolve(item)
})
}).catch((err) => {
logger.error(err);
opResolve(item)
})
}).catch((err) => {
logger.error(err);
opResolve(item)
})
})
})
}
subSyncPromises.push(() => {
return new Promise((resolve) => {
this.processAttributes(item.custom_attributes, item.configurable_options || []).then(res => {
item.attributes_metadata = res
item.custom_attributes = null
resolve(item)
})
})
});
// CATEGORIES SYNC
subSyncPromises.push(() => {
return new Promise((resolve, reject) => {
logger.info(`Product sub-stage 6: Getting product categories for ${item.sku}`);
const key = util.format(CacheKeys.CACHE_KEY_PRODUCT_CATEGORIES, item.sku); // store under SKU of the product the categories assigned
if(this.category_sync) {
item.category = new Array();
const catBinder = (categories) => {
let catPromises = new Array();
for (let catId of categories) {
catPromises.push(
new Promise((resolve, reject) => {
let cat = this.cache.get(util.format(CacheKeys.CACHE_KEY_CATEGORY, catId), (err, serializedCat) => {
let cat = JSON.parse(serializedCat); // category object
if (cat != null) {
resolve({
category_id: cat.id,
name: cat.name,
slug: cat.slug,
path: cat.url_path
})
} else {
resolve({
category_id: catId
});
}
});
})
);
}
Promise.all(catPromises).then((values) => {
if(this.category_sync) // TODO: refactor the code above to not get cache categorylinks when no category_sync required
item.category = values; // here we get configurable options
if (this.config.seo.useUrlDispatcher) {
item.url_path = this.config.seo.productUrlPathMapper(item)
}
resolve(item)
});
}
if (item.category_ids && Array.isArray(item.category_ids) && item.category_ids.length > 0) {
const catIdsArray = item.category_ids.map(item => { return parseInt(item)})
logger.info(`Using category_ids binding for ${item.sku}: ${catIdsArray}`)
catBinder(catIdsArray)
} else {
this.cache.smembers(key, (err, categories) => {
if (categories == null) {
resolve(item);
}
else {
catBinder(categories)
}
})
}
}
})})
serial(subSyncPromises).then((res) => {
logger.info(`Product sub-stages done for ${item.sku}`)
return done(item) // all subpromises return refernce to the product
}).catch(err => {
logger.error(err);
return done(item)
});
});
}
/**
* Process video data to provide the proper
* provider and attributes.
* Currently supports YouTube and Vimeo
*
* @param {Object} mediaItem
*/
computeVideoData(mediaItem) {
let videoData = null;
if (mediaItem.extension_attributes && mediaItem.extension_attributes.video_content) {
let videoId = null,
type = null,
youtubeRegex = /^(?:https?:\/\/)?(?:www\.)?(?:youtu\.be\/|youtube\.com\/(?:embed\/|v\/|watch\?v=|watch\?.+&v=))((\w|-){11})(?:\S+)?$/,
vimeoRegex = new RegExp(['https?:\\/\\/(?:www\\.|player\\.)?vimeo.com\\/(?:channels\\/(?:\\w+\\/)',
'?|groups\\/([^\\/]*)\\/videos\\/|album\\/(\\d+)\\/video\\/|video\\/|)(\\d+)(?:$|\\/|\\?)'
].join(''));
if (mediaItem.extension_attributes.video_content.video_url.match(youtubeRegex)) {
videoId = RegExp.$1
type = 'youtube'
} else if (mediaItem.extension_attributes.video_content.video_url.match(vimeoRegex)) {
videoId = RegExp.$3
type = 'vimeo'
}
videoData = {
url: mediaItem.extension_attributes.video_content.video_url,
title: mediaItem.extension_attributes.video_content.video_title,
desc: mediaItem.extension_attributes.video_content.video_description,
meta: mediaItem.extension_attributes.video_content.video_metadata,
video_id: videoId,
type: type
}
}
return videoData;
}
/**
* We're transorming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
let prices = new Array();
/*for (let priceTag of item.tier_prices) {
prices.push({
"price": priceTag.value,
"original_price": priceTag.original_price,
"customer_group_id": priceTag.customerGroupId,
"qty": priceTag.qty
});
}*/
if (this.config.vuestorefront && this.config.vuestorefront.invalidateCache) {
request(this.config.vuestorefront.invalidateCacheUrl + 'P' + item.id, {}, (err, res, body) => {
if (err) { return console.error(err); }
try {
if (body && JSON.parse(body).code !== 200) console.log(body);
} catch (e) {
return console.error('Invalid Cache Invalidation response format', e)
}
});
}
let resultItem = Object.assign(item, {
// "price": prices, // ES stores prices differently
// TODO: HOW TO GET product stock from Magento API call for product?
});
return resultItem;
}
}
module.exports = ProductAdapter;
================================================
FILE: src/adapters/magento/productcategories.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
let queue = require('queue');
let util = require('util');
let q = queue({ concurrency: 4 });
const ProductCategoriesModes = {
SEPARATE_ELASTICSEARCH: 1,
UPDATE_CATEGORY: 2,
SEPARATE_REDIS: 3
}
const CacheKeys = require('./cache_keys');
/**
* This adapter retrieves and stores all product / category links from Magento2
*/
class ProductcategoriesAdapter extends AbstractMagentoAdapter {
constructor(config) {
super(config);
this.update_document = false; // this adapter works on categories but doesn't update them itself but is focused on category links instead!
this._productHisto = new Set();
this.mode = ProductCategoriesModes.SEPARATE_REDIS;
this.catLinkQueue = new Array();
}
getEntityType() {
return 'productcategories';
}
getName() {
return 'adapters/magento/ProductcategoryAdapter';
}
getSourceData(context) {
return this.api.categories.list();
}
getLabel(source_item) {
return `[(${source_item.id}) ${source_item.name}]`;
}
/**
* Process category link product/category
* @param {Object} result
*/
_storeCatLinks(item, result) {
let index = 0;
let length = result.length;
switch (this.mode) {
case ProductCategoriesModes.SEPARATE_ELASTICSEARCH: {
for (let catLink of result) {
// logger.debug('(' +index +'/' + length + ') Storing categoryproduct link for ' + catLink.sku +' - ' + catLink.category_id);
catLink.id = catLink.category_id * MAX_PRODUCTS_IN_CAT + index;
index++;
catLink = this.normalizeDocumentFormat(catLink);
}
logger.debug('Performing bulk update...');
this.db.updateDocumentBulk('productcategories', result); // TODO: add support for BULK operations and DELETE
}
case ProductCategoriesModes.UPDATE_CATEGORY: {
// TODO: update products to set valid category
item.products = result;
this.db.updateDocument('category', item);
logger.debug(`Updating category object for ${item.id}`);
}
default: { // update in redis = ProductCategoriesModes.SEPARATE_REDIS
logger.debug(`Storing category assigments in REDIS cache for ${item.id}`);
for (let catLink of result) {
const key = util.format(CacheKeys.CACHE_KEY_PRODUCT_CATEGORIES_TEMPORARY, catLink.sku); // store under SKU of the product the categories assigned
this.cache.sadd(key, catLink.category_id); // add categories to sets
this._productHisto.add(catLink.sku);
}
}
}
logger.info(`The task count still is = ${this.tasks_count}`);
}
/**
* Get the product category links for this specific category and update the products
* @param {Object} item
*/
preProcessItem(item) {
return new Promise((done, reject) => {
// q.push(() => {
this.api.categoryProducts.list(item.id).then((result) => {
this._storeCatLinks(item, result);
done(item);
}).catch((err) => {
logger.error(err);
done(item);
});
// });
});
}
/**
* Default done callback called after all main items are processed by processItems
*/
defaultDoneCallback() {
logger.info('Renaming the cache key to production! ');
for(let sku of this._productHisto){
const origKey = util.format(CacheKeys.CACHE_KEY_PRODUCT_CATEGORIES_TEMPORARY, sku);
const destKey = util.format(CacheKeys.CACHE_KEY_PRODUCT_CATEGORIES, sku);
// logger.debug(util.format('Moving %s to %s', origKey, destKey));
this.cache.rename(origKey, destKey);
}
logger.info('Publishing cache keys to production finished!');
/* q.start((err) => {
if (err) throw err
logger.info('all done:', results)
});*/
}
isFederated() {
return false;
}
/**
* We're transorming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
return item;
}
}
module.exports = ProductcategoriesAdapter;
================================================
FILE: src/adapters/magento/review.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
class ReviewAdapter extends AbstractMagentoAdapter {
constructor(config) {
super(config);
this.use_paging = false;
}
getEntityType() {
return 'review';
}
getName() {
return 'adapters/magento/ReviewAdapter';
}
getSourceData(context) {
if (this.use_paging) {
return this.api.reviews.list('&searchCriteria[currentPage]=' + this.page + '&searchCriteria[pageSize]=' + this.page_size + (query ? '&' + query : '')).catch((err) => {
throw new Error(err);
});
}
return this.api.reviews.list().catch((err) => {
throw new Error(err);
});
}
/**
* Regarding Magento2 api docs and reality
* we do have an exception here that items aren't listed straight
* in the response but under "items" key */
prepareItems(items) {
if (!items) {
return items;
}
if (items.total_count) {
this.total_count = items.total_count;
}
if (this.use_paging) {
this.page_count = Math.ceil(this.total_count / this.page_size);
}
if (items.items) {
items = items.items; // this is an exceptional behavior for Magento2 api for lists
}
return items;
}
isFederated() {
return false;
}
preProcessItem(item) {
logger.debug(item);
//
return new Promise((done, reject) => {
if (item) {
item.product_id = item.entity_pk_value;
delete item.entity_pk_value;
logger.debug(`Review ${item.id}`);
}
return done(item);
});
}
/**
* We're transorming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
return item;
}
}
module.exports = ReviewAdapter;
================================================
FILE: src/adapters/magento/taxrule.js
================================================
'use strict';
let AbstractMagentoAdapter = require('./abstract');
class TaxruleAdapter extends AbstractMagentoAdapter {
getEntityType() {
return 'taxrule';
}
getName() {
return 'adapters/magento/TaxrulesAdapter';
}
getSourceData(context) {
return this.api.taxRules.list();
}
getLabel(source_item) {
return `[(${source_item.id}) ${source_item.code}]`;
}
isFederated() {
return false;
}
preProcessItem(item) {
return new Promise((done, reject) => {
// TODO get tax rates for this tax rule
let subPromises = []
item.rates = []
logger.info(item)
for (let ruleId of item.tax_rate_ids) {
subPromises.push(new Promise((resolve, reject) => {
this.api.taxRates.list(ruleId).then(function(result) {
result.rate = parseFloat(result.rate)
item.rates.push(result)
resolve (result)
})
}))
}
Promise.all(subPromises).then(function(results) {
logger.info(`Rates downloaded for ${item.code}`)
logger.info(item)
return done(item);
})
});
}
prepareItems(items) {
if(!items)
return null;
this.total_count = items.total_count;
return items.items;
}
/**
* We're transorming the data structure of item to be compliant with Smile.fr Elastic Search Suite
* @param {object} item document to be updated in elastic search
*/
normalizeDocumentFormat(item) {
return item;
}
}
module.exports = TaxruleAdapter;
================================================
FILE: src/adapters/nosql/abstract.js
================================================
'use strict';
class AbstractNosqlAdapter{
validateConfig(config){
if(!config['db']['url'])
throw Error('db.url must be set up in config');
}
constructor(app_config){
this.config = app_config;
this.db = null;
this.validateConfig(this.config);
this.updateDocument.bind(this);
}
/**
* Get where query to identify the base record
* @param {Object} source_item
*/
getWhereQuery(source_item){
return { id: source_item.id };
}
/**
* Close the nosql database connection - abstract to the driver
*/
close(){
throw new Error('close needs implementation!');
}
/**
* Update single document in database
* @param {object} item document to be updated in database
*/
updateDocument(collectionName, item) {
throw new Error('updateDocument needs implementation!');
}
/**
* Get documents
* @param collectionName collection name
* @param query query object
*/
getDocuments(collectionName, query) {
throw new Error('getDocum ent needs implementation!');
}
/**
* Update multiple documents in database
* @param {array} items to be updated
*/
updateDocumentBulk(collectionName, items){
throw new Error('updateDocumentBulk needs implementation!');
}
/**
* Remove records other than <record>.tsk = "transactionKey"
* @param {String} collectionName
* @param {int} transactionKey transaction key - which is usualy a timestamp
*/
cleanupByTransactionkey(collectionName, transactionKey){
throw new Error('cleanupByTransactionkey needs implementation!');
}
/**
* Connect / prepare driver
*/
connect (done){
throw new Error('connect needs implementation!');
}
}
module.exports = AbstractNosqlAdapter;
================================================
FILE: src/adapters/nosql/elasticsearch.js
================================================
'use strict';
const AbstractNosqlAdapter = require('./abstract');
const elasticsearch = require('@elastic/elasticsearch');
const AgentKeepAlive = require('agentkeepalive');
const AgentKeepAliveHttps = require('agentkeepalive').HttpsAgent;
class ElasticsearchAdapter extends AbstractNosqlAdapter {
validateConfig(config) {
if (!config['db']['url'])
throw Error('db.url must be set up in config');
if (!config['db']['indexName'])
throw Error('db.indexName must be set up in config');
}
constructor(app_config) {
super(app_config);
this.config = app_config;
this.db = null;
this.validateConfig(this.config);
logger.debug('Elasticsearch module initialized!');
this.updateDocument.bind(this);
}
/**
* Get physical Elastic index name; since 7.x we're adding an entity name to get real index: vue_storefront_catalog_product, vue_storefront_catalog_category and so on
* @param {*} baseIndexName
* @param {*} config
*/
getPhysicalIndexName(collectionName, config) {
if (parseInt(config.elasticsearch.apiVersion) >= 6) {
return `${config.db.indexName}_${collectionName}`
} else {
return config.db.indexName
}
}
/**
* Get physical Elastic type name; since 7.x index can have one type _doc
* @param {*} baseIndexName
* @param {*} config
*/
getPhysicalTypeName(collectionName, config) {
if (parseInt(config.elasticsearch.apiVersion) >= 6) {
return `_doc`
} else {
return collectionName
}
}
/**
* Close the nosql database connection - abstract to the driver
*/
close() { // switched to global singleton
//this.db.close();
}
/**
* Get documents
* @param collectionName collection name
* @param query query object
*/
getDocuments(collectionName, queryBody) {
return new Promise((resolve, reject) => {
const searchQueryBody = {
index: this.getPhysicalIndexName(collectionName, this.config),
body: queryBody
}
if (parseInt(this.config.elasticsearch.apiVersion) < 6)
searchQueryBody.type = this.getPhysicalTypeName(collectionName, this.config)
this.db.search(searchQueryBody, function (error, { body: response }) {
if (error) reject(error);
if (response.hits && response.hits.hits) {
resolve(response.hits.hits.map(h => h._source))
} else {
reject(new Error('Invalid Elastic response'))
}
});
})
}
/**
* Update single document in database
* @param {object} item document to be updated in database
*/
updateDocument(collectionName, item) {
const itemtbu = item;
const updateRequestBody = {
index: this.getPhysicalIndexName(collectionName, this.config),
id: item.id,
body: {
// put the partial document under the `doc` key
upsert: itemtbu,
doc: itemtbu
}
}
if (parseInt(this.config.elasticsearch.apiVersion) < 6)
updateRequestBody.type = this.getPhysicalTypeName(collectionName, this.config)
this.db.update(updateRequestBody, function (error, response) {
if (error)
throw new Error(error);
});
}
/**
* Remove records other than <record>.tsk = "transactionKey"
* @param {String} collectionName
* @param {int} transactionKey transaction key - which is usually a timestamp
*/
cleanupByTransactionkey(collectionName, transactionKey) {
if (transactionKey) {
const query = {
index: this.getPhysicalIndexName(collectionName, this.config),
conflicts: 'proceed',
body: {
query: {
bool: {
must_not: {
term: { tsk: transactionKey }
}
}
}
}
};
if (parseInt(this.config.elasticsearch.apiVersion) < 6)
query.type = this.getPhysicalTypeName(collectionName, this.config)
this.db.deleteByQuery(query, function (error, response) {
if (error) throw new Error(error);
});
}
}
/**
* Update multiple documents in database
* @param {array} items to be updated
*/
updateDocumentBulk(collectionName, items) {
let requests = new Array();
let index = 0;
let bulkSize = 500;
for (let doc of items) {
const query = {
_index: this.getPhysicalIndexName(collectionName, this.config),
_id: doc.id,
};
if (parseInt(this.config.elasticsearch.apiVersion) < 6)
query.type = this.getPhysicalTypeName(collectionName, this.config)
requests.push({
update: query
});
requests.push({
// put the partial document under the `doc` key
doc: doc,
"doc_as_upsert": true
});
if ((index % bulkSize) == 0) {
logger.debug('Splitting bulk query ' + index);
this.db.bulk({
body: requests
}, function (error, response) {
if (error)
throw new Error(error);
});
requests = new Array();
}
index++;
}
}
/**
* Connect / prepare driver
* @param {Function} done callback to be called after connection is established
*/
connect(done) {
if (!global.es) {
this.db = new elasticsearch.Client({
node: this.config.db.url,
log: 'debug',
apiVersion: this.config.elasticsearch.apiVersion,
maxRetries: 10,
keepAlive: true,
maxSockets: 10,
minSockets: 10,
requestTimeout: 1800000,
createNodeAgent: function (connection, config) {
if (connection.useSsl) {
return new AgentKeepAliveHttps(connection.makeAgentConfig(config));
}
return new AgentKeepAlive(connection.makeAgentConfig(config));
}
});
global.es = this.db;
} else
this.db = global.es;
done(this.db);
}
}
module.exports = ElasticsearchAdapter;
================================================
FILE: src/api/routes/magento.js
================================================
'use strict';
let express = require('express');
let kue = require('kue');
let config = require('../../config');
let AdapterFactory = require('../../adapters/factory');
let factory = new AdapterFactory(config);
let router = express.Router();
router.post('/products/update', function(req, res) { // TODO: add api key middleware
let skus_array = req.body.sku;
console.log('Incoming pull request for', skus_array)
if(skus_array.length > 0){
let queue = kue.createQueue(Object.assign(config.kue, { redis: config.redis }));
queue.createJob('product', { skus: skus_array, adapter: 'magento' }).save();
res.json({ status: 'done', message: 'Products ' + skus_array + ' scheduled to be refreshed'});
} else {
res.json({ status: 'error', message: 'Please provide product SKU separated by comma'});
}
});
module.exports = router;
================================================
FILE: src/cli.js
================================================
'use strict';
const program = require('commander');
const fs = require('fs');
const path = require('path');
let AdapterFactory = require('./adapters/factory');
const TIME_TO_EXIT = process.env.TIME_TO_EXIT ? process.env.TIME_TO_EXIT : 30000; // wait 30s before quiting after task is done
let config = require('./config');
let logger = require('./log');
let factory = new AdapterFactory(config);
const jsonFile = require('jsonfile')
const INDEX_META_PATH = process.env.INDEX_META_PATH ? path.join('tmp', process.env.INDEX_META_PATH) : path.join('tmp', '.lastIndex.json')
let kue = require('kue');
let queue = kue.createQueue(Object.assign(config.kue, { redis: config.redis }));
const _handleBoolParam = (value) => {
return JSON.parse(value) // simple way to handle all the '0', '1', 'true', true, false ...
}
const reindexAttributes = (adapterName, removeNonExistent) => {
removeNonExistent = _handleBoolParam(removeNonExistent)
return new Promise((resolve, reject) => {
let adapter = factory.getAdapter(adapterName, 'attribute');
let tsk = new Date().getTime();
adapter.run({
transaction_key: tsk,
done_callback: () => {
if (removeNonExistent) {
adapter.cleanUp(tsk);
}
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
resolve();
}
});
})
}
const reindexReviews = (adapterName, removeNonExistent) => {
removeNonExistent = _handleBoolParam(removeNonExistent)
return new Promise((resolve, reject) => {
let adapter = factory.getAdapter(adapterName, 'review');
let tsk = new Date().getTime();
adapter.cleanUp(tsk);
adapter.run({
transaction_key: tsk,
done_callback: () => {
if (removeNonExistent) {
adapter.cleanUp(tsk);
}
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
resolve();
}
});
});
}
/**
* Re-index cms blocks
*/
const reindexBlocks = (adapterName, removeNonExistent) => {
removeNonExistent = _handleBoolParam(removeNonExistent)
return new Promise((resolve, reject) => {
let adapter = factory.getAdapter(adapterName, 'cms_block');
let tsk = new Date().getTime();
adapter.run({
transaction_key: tsk,
done_callback: () => {
if (removeNonExistent) {
adapter.cleanUp(tsk);
}
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
resolve();
}
})
})
}
/**
* Re-index cms pages
*/
const reindexPages = (adapterName, removeNonExistent) => {
removeNonExistent = _handleBoolParam(removeNonExistent)
return new Promise((resolve, reject) => {
let adapter = factory.getAdapter(adapterName, 'cms_page');
let tsk = new Date().getTime();
adapter.run({
transaction_key: tsk,
done_callback: () => {
if (removeNonExistent) {
adapter.cleanUp(tsk);
}
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
resolve();
}
})
})
}
const reindexCategories = (adapterName, removeNonExistent, extendedCategories, generateUniqueUrlKeys) => {
removeNonExistent = _handleBoolParam(removeNonExistent)
extendedCategories = _handleBoolParam(extendedCategories)
generateUniqueUrlKeys = (_handleBoolParam(generateUniqueUrlKeys))
return new Promise((resolve, reject) => {
let adapter = factory.getAdapter(adapterName, 'category');
let tsk = new Date().getTime();
adapter.run({
transaction_key: tsk,
extendedCategories: extendedCategories,
generateUniqueUrlKeys: generateUniqueUrlKeys,
done_callback: () => {
if (removeNonExistent) {
adapter.cleanUp(tsk);
}
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
resolve();
}
});
});
}
const reindexTaxRules = (adapterName, removeNonExistent) => {
removeNonExistent = _handleBoolParam(removeNonExistent)
return new Promise((resolve, reject) => {
let adapter = factory.getAdapter(adapterName, 'taxrule');
let tsk = new Date().getTime();
adapter.run({
transaction_key: tsk,
done_callback: () => {
if (removeNonExistent) {
adapter.cleanUp(tsk);
}
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
resolve();
}
});
});
}
const reindexProductCategories = (adapterName) => {
return new Promise((resolve, reject) => {
let adapter = factory.getAdapter(adapterName, 'productcategories');
adapter.run({
done_callback: () => {
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
resolve();
}
});
});
}
function cleanup(adapterName, cleanupType, transactionKey) {
let adapter = factory.getAdapter(adapterName, cleanupType);
let tsk = transactionKey;
if (tsk) {
logger.info('Cleaning up for TRANSACTION KEY = ' + tsk);
adapter.connect
adapter.cleanUp(tsk);
} else {
logger.error('No "transactionKey" given as a parameter');
}
}
function reindexProducts(adapterName, removeNonExistent, partitions, partitionSize, initQueue, skus, updatedAfter = null, page = null) {
removeNonExistent = _handleBoolParam(removeNonExistent)
initQueue = _handleBoolParam(initQueue)
let adapter = factory.getAdapter(adapterName, 'product');
if (updatedAfter) {
logger.info('Delta indexer started for', updatedAfter)
}
let tsk = new Date().getTime();
if (partitions > 1 && adapter.isFederated()) { // standard case
let partition_count = partitions;
logger.info(`Running in MPM (Multi Process Mode) with partitions count = ${partition_count}`);
adapter.getTotalCount({ updated_after: updatedAfter }).then((result) => {
let total_count = result.total_count;
let page_size = partitionSize;
let page_count = Math.ceil(total_count / page_size);
let transaction_key = new Date().getTime();
if (initQueue) {
logger.info('Propagating job queue... ');
for (let i = 1; i <= page_count; i++) {
logger.debug(`Adding job for: ${i} / ${page_count}, page_size = ${page_size}`);
queue.createJob('products', { page_size: page_size, page: i, updated_after: updatedAfter }).save();
}
} else {
logger.info('Not propagating queue - only worker mode!');
}
// TODO: separate the execution part to run in multi-tenant env
queue.process('products', partition_count, (job, done) => {
let adapter = factory.getAdapter(adapterName, 'product');
if (job && job.data.page && job.data.page_size) {
logger.info(`Processing job: ${job.data.page}`);
adapter.run({
transaction_key: transaction_key,
page_size: job.data.page_size,
page: job.data.page,
parent_sync: job.data.updatedAfter !== null,
updated_after: job.data.updatedAfter,
done_callback: () => {
logger.info('Task done!');
return done();
}
});
} else return done();
});
if (initQueue) { // if this is not true it meant that process is runing to process the queue in the loop and shouldnt be "killed"
setInterval(() => {
queue.inactiveCount((err, total) => { // others are activeCount, completeCount, failedCount, delayedCount
if (total == 0) {
if (removeNonExistent) {
logger.info('CleaningUp products!');
let adapter = factory.getAdapter(adapterName, 'product');
adapter.cleanUp(transaction_key);
}
logger.info('Queue processed. Exiting!');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
}
});
}, 2000);
}
});
} else {
logger.info('Running in SPM (Single Process Mode)');
let context = {
page: page !== null ? parseInt(page) : null,
page_size: partitionSize,
use_paging: true,
updated_after: updatedAfter,
transaction_key: tsk,
parent_sync: updatedAfter !== null,
done_callback: () => {
if (removeNonExistent) {
adapter.cleanUp(tsk);
}
logger.info('Task done! Exiting in 30s...');
setTimeout(process.exit, TIME_TO_EXIT); // let ES commit all changes made
}
};
if (page!== null) logger.info('Current page is: ', page, partitionSize)
if (skus) {
context.skus = skus.split(','); // update individual producs
context.parent_sync = true
}
adapter.run(context);
}
}
function fullReindex(adapterName, removeNonExistent, partitions, partitionSize, initQueue, skus, extendedCategories, generateUniqueUrlKeys) {
// The sequence is important because commands operate on some cache resources - especially for product/category assignments
Promise.all([
reindexAttributes(adapterName, removeNonExistent), // 0. It stores attributes in redis cache
reindexTaxRules(adapterName, removeNonExistent), // 1. It indexes the taxRules
reindexCategories(adapterName, removeNonExistent, extendedCategories, generateUniqueUrlKeys), //2. It stores categories in redis cache
reindexProductCategories(adapterName) // 3. It stores product/cateogry links in redis cache
]).then((results) => {
logger.info('Starting full products reindex!');
reindexProducts(adapterName, removeNonExistent, partitions, partitionSize, initQueue, skus); //4. It indexes all the products
}).catch((err) => {
logger.error(err);
process.exit(1)
});
}
/**
* Run worker listening to "product" command on KUE queue
*/
function runProductsworker(adapterName, partitions) {
logger.info('Starting `productsworker` worker. Waiting for jobs ...');
let partition_count = partitions;
// TODO: separte the execution part to run in multi-tenant env
queue.process('product', partition_count, (job, done) => {
if (job && job.data.skus && Array.isArray(job.data.skus)) {
logger.info('Starting product pull job for ' + job.data.skus.join(','));
let adapter = factory.getAdapter(job.data.adapter ? job.data.adapter : adapterName, 'product');
adapter.run({
skus: job.data.skus,
parent_sync: true,
done_callback: () => {
logger.info('Task done!');
return done();
}
});
} else return done();
});
}
program
.command('attributes')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.action(async (cmd) => {
await reindexAttributes(cmd.adapter, cmd.removeNonExistent);
});
program
.command('categories')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.option('--extendedCategories <extendedCategories>', 'extended categories import', true)
.option('--generateUniqueUrlKeys <generateUniqueUrlKeys>', 'make sure that category url keys are uniqe', true)
.action(async (cmd) => {
await reindexCategories(cmd.adapter, cmd.removeNonExistent, cmd.extendedCategories, cmd.generateUniqueUrlKeys);
});
program
.command('cleanup')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--cleanupType <cleanupType>', 'type of the entity to clean up: product|category', 'product')
.option('--transactionKey <transactionKey>', 'transaction key', 0)
.action((cmd) => {
cleanup(cmd.adapter, cmd.cleanupType, cmd.transactionKey);
});
program
.command('fullreindex')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--partitions <partitions>', 'number of partitions', 1)
.option('--partitionSize <partitionSize>', 'size of the partitions', 50)
.option('--initQueue <initQueue>', 'use the queue', true)
.option('--skus <skus>', 'comma delimited list of SKUs to fetch fresh informations from', '')
.option('--extendedCategories <extendedCategories>', 'extended categories import', true)
.option('--generateUniqueUrlKeys <generateUniqueUrlKeys>', 'generate unique url_keys', true)
.action((cmd) => {
fullReindex(cmd.adapter, true, cmd.partitions, cmd.partitionSize, cmd.initQueue, cmd.skus, cmd.extendedCategories, cmd.generateUniqueUrlKeys);
});
program
.command('productcategories')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.action((cmd) => {
reindexProductCategories(cmd.adapter);
});
program
.command('products')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--partitions <partitions>', 'number of partitions', 1)
.option('--partitionSize <partitionSize>', 'size of the partitions', 50)
.option('--initQueue <initQueue>', 'use the queue', true)
.option('--skus <skus>', 'comma delimited list of SKUs to fetch fresh informations from', '')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.option('--updatedAfter <updatedAfter>', 'timestamp to start the synchronization from', '')
.option('--page <page>', 'start from specific page', null)
.action((cmd) => {
if (cmd.updatedAfter) {
reindexProducts(cmd.adapter, cmd.removeNonExistent, cmd.partitions, cmd.partitionSize, cmd.initQueue, cmd.skus, new Date(cmd.updatedAfter), cmd.page);
} else {
reindexProducts(cmd.adapter, cmd.removeNonExistent, cmd.partitions, cmd.partitionSize, cmd.initQueue, cmd.skus, null, cmd.page);
}
});
program
.command('productsdelta')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--partitions <partitions>', 'number of partitions', 1)
.option('--partitionSize <partitionSize>', 'size of the partitions', 50)
.option('--initQueue <initQueue>', 'use the queue', true)
.option('--skus <skus>', 'comma delimited list of SKUs to fetch fresh informations from', '')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.action((cmd) => {
let indexMeta = { lastIndexDate: new Date() }
let updatedAfter = null
try {
// Make sure so the temporary folder exists.
if (!fs.existsSync('tmp')) {
fs.mkdirSync('tmp')
}
indexMeta = jsonFile.readFileSync(INDEX_META_PATH)
updatedAfter = new Date(indexMeta.lastIndexDate)
} catch (err) {
console.log('Seems like first time run!')
updatedAfter = null // full reindex
}
reindexProducts(cmd.adapter, cmd.removeNonExistent, cmd.partitions, cmd.partitionSize, cmd.initQueue, cmd.skus, updatedAfter);
try {
indexMeta.lastIndexDate = new Date()
jsonFile.writeFile(INDEX_META_PATH, indexMeta)
} catch (err) {
console.log('Error writing index meta!', err)
}
})
program
.command('productsworker')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--partitions <partitions>', 'number of partitions', 1)
.action((cmd) => {
runProductsworker(cmd.adapter, cmd.partitions);
})
program
.command('reviews')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.action(async (cmd) => {
await reindexReviews(cmd.adapter, cmd.removeNonExistent);
})
program
.command('taxrule')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.action(async (cmd) => {
await reindexTaxRules(cmd.adapter, cmd.removeNonExistent);
})
/**
* Sync cms blocks
*/
program
.command('blocks')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.action(async (cmd) => {
await reindexBlocks(cmd.adapter, cmd.removeNonExistent);
})
program
.command('pages')
.option('--adapter <adapter>', 'name of the adapter', 'magento')
.option('--removeNonExistent <removeNonExistent>', 'remove non existent products', false)
.action(async (cmd) => {
await reindexPages(cmd.adapter, cmd.removeNonExistent);
})
program
.on('command:*', () => {
console.error('Invalid command: %s\nSee --help for a list of available commands.', program.args.join(' '));
process.exit(1);
});
program
.parse(process.argv);
process
.on('unhandledRejection', (reason, p) => {
logger.error('Unhandled Rejection at: Promise', p, 'reason:', reason);
// application specific logging, throwing an error, or other logic here
});
================================================
FILE: src/config.js
================================================
const _slugify = require('./helpers/slugify')
module.exports = {
seo: {
useUrlDispatcher: JSON.parse(process.env.SEO_USE_URL_DISPATCHER || true),
productUrlPathMapper: (product) => {
let destPath = ''
if (product.category && product.category.length > 0) {
for (let i = 0; i < product.category.length; i++) {
if (typeof product.category[i].name !== 'undefined') {
const firstValidCat = product.category[i]
destPath = (firstValidCat.path ? (firstValidCat.path) : _slugify(firstValidCat.name)) + '/' + (product.slug ? product.slug : _slugify(product.name + '-' + product.id))
break
}
}
}
if (destPath === '') {
destPath = (product.slug ? product.slug : _slugify(product.name + '-' + product.id))
}
destPath += '.html'
console.log('Dest. product path = ', destPath)
return destPath
},
categoryUrlPathMapper: (category) => {
const destSlug = (category.url_path ? category.url_path + '/': '') + category.url_key
console.log('Dest. cat path = ', destSlug)
return destSlug
},
},
magento: {
url: process.env.MAGENTO_URL || 'http://magento2.demo-1.divante.pl/rest/',
consumerKey: process.env.MAGENTO_CONSUMER_KEY || 'alva6h6hku9qxrpfe02c2jalopx7od1q',
consumerSecret: process.env.MAGENTO_CONSUMER_SECRET || '9tgfpgoojlx9tfy21b8kw7ssfu2aynpm',
accessToken: process.env.MAGENTO_ACCESS_TOKEN || 'rw5w0si9imbu45h3m9hkyrfr4gjina8q',
accessTokenSecret: process.env.MAGENTO_ACCESS_TOKEN_SECRET || '00y9dl4vpxgcef3gn5mntbxtylowjcc9',
storeId: process.env.MAGENTO_STORE_ID || 1,
currencyCode: process.env.MAGENTO_CURRENCY_CODE || 'USD',
msi: { enabled: process.env.MAGENTO_MSI_ENABLED || false, stockId: process.env.MAGENTO_MSI_STOCK_ID || 1 }
},
vuestorefront: {
invalidateCache: JSON.parse(typeof process.env.VS_INVALIDATE_CACHE === 'undefined' ? false : process.env.VS_INVALIDATE_CACHE),
invalidateCacheUrl: process.env.VS_INVALIDATE_CACHE_URL || 'http://localhost:3000/invalidate?key=aeSu7aip&tag='
},
product: {
expandConfigurableFilters: ['manufacturer'],
synchronizeCatalogSpecialPrices: process.env.PRODUCTS_SPECIAL_PRICES || true,
renderCatalogRegularPrices: process.env.PRODUCTS_RENDER_PRICES || false,
excludeDisabledProducts: process.env.PRODUCTS_EXCLUDE_DISABLED || false
},
kue: {}, // default KUE config works on local redis instance. See KUE docs for non standard redis connections
db: {
driver: 'elasticsearch',
url: process.env.DATABASE_URL || 'http://localhost:9200',
indexName: process.env.INDEX_NAME || 'vue_storefront_catalog'
},
elasticsearch: {
apiVersion: process.env.ELASTICSEARCH_API_VERSION || '5.6'
},
redis: {
host: process.env.REDIS_HOST || '127.0.0.1',
port: process.env.REDIS_PORT || 6379,
auth: process.env.REDIS_AUTH || false,
db: process.env.REDIS_DB || 0
},
passport: {
jwtSecret: "MyS3cr3tK3Y",
jwtSession: {
session: false
}
}
}
================================================
FILE: src/helpers/slugify.js
================================================
const removeAccents = require('remove-accents').remove
/**
* Create slugify -> "create-slugify" permalink of text
* @param {String} text
* @returns {String}
*/
function slugify (text) {
// to be sure that text is String
text = text.toString()
// remove regional characters
text = removeAccents(text)
return text
.toLowerCase()
.replace(/\s+/g, '-') // Replace spaces with -
.replace(/&/g, '-and-') // Replace & with 'and'
.replace(/[^\w-]+/g, '') // Remove all non-word chars
.replace(/--+/g, '-') // Replace multiple - with single -
}
module.exports = slugify
================================================
FILE: src/log.js
================================================
var winston = require('winston');
winston.emitErrs = true;
if(!global.logger) {
global.logger = new winston.Logger({
transports: [
new winston.transports.Console({
level: 'debug',
handleExceptions: false,
json: false,
prettyPrint: true,
colorize: true,
timestamp: true
})
],
exitOnError: false
});
}
module.exports = global.logger;
================================================
FILE: src/test_by_sku.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js products --partitions=1 --skus=WS01-XS-BLACK
================================================
FILE: src/test_categoryextended.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
================================================
FILE: src/test_fullreindex.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export SEO_USE_URL_DISPATCHER=1
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js blocks
node --harmony cli.js pages
node --harmony cli.js reviews
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
node --harmony cli.js productcategories
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
================================================
FILE: src/test_fullreindex_de.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export SEO_USE_URL_DISPATCHER=1
echo 'Italian store - de'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/de
export INDEX_NAME=vue_storefront_catalog_de
node --harmony cli.js blocks
node --harmony cli.js pages
node --harmony cli.js reviews
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
node --harmony cli.js productcategories
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
================================================
FILE: src/test_fullreindex_it.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export SEO_USE_URL_DISPATCHER=1
echo 'Italian store - it'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/it
export INDEX_NAME=vue_storefront_catalog_it
node --harmony cli.js blocks
node --harmony cli.js pages
node --harmony cli.js reviews
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
node --harmony cli.js productcategories
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
================================================
FILE: src/test_fullreindex_multiprocess.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js blocks
node --harmony cli.js pages
node --harmony cli.js reviews
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
node --harmony cli.js productcategories
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --partitions=4
================================================
FILE: src/test_multistore.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export SEO_USE_URL_DISPATCHER=1
echo 'German store - de'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/de
export INDEX_NAME=vue_storefront_catalog_de
node --harmony cli.js blocks
node --harmony cli.js pages
node --harmony cli.js reviews
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
node --harmony cli.js productcategories
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
echo 'Italian store - it'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/it
export INDEX_NAME=vue_storefront_catalog_it
node --harmony cli.js blocks
node --harmony cli.js pages
node --harmony cli.js reviews
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
node --harmony cli.js productcategories
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
export INDEX_NAME=vue_storefront_catalog
node --harmony cli.js blocks
node --harmony cli.js pages
node --harmony cli.js reviews
node --harmony cli.js categories --removeNonExistent=true --extendedCategories=true
node --harmony cli.js productcategories
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
================================================
FILE: src/test_product.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js products --removeNonExistent=true --partitions=1
================================================
FILE: src/test_product_delta.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js productsdelta
================================================
FILE: src/test_product_msi.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_MSI_ENABLED=true
export MAGENTO_MSI_STOCK_ID=1
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js products --removeNonExistent=true --partitions=1
================================================
FILE: src/test_product_worker.sh
================================================
export TIME_TO_EXIT=2000
export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
export PRODUCTS_SPECIAL_PRICES=true
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
node --harmony cli.js productsworker
================================================
FILE: src/tmp/.gitignore
================================================
*.json
================================================
FILE: src/webapi.js
================================================
'use strict';
/**
* Webhook API to add specific products or categories to be synchronized by the service
*/
let logger = require('./log');
var express = require('express');
var app = express();
var body_parser = require('body-parser');
// configure app to use bodyParser()
// this will let us get the data from a POST
app.use(body_parser.urlencoded({ extended: true }));
app.use(body_parser.json());
var port = process.env.PORT || 8080; // set our port
app.use('/magento', require('./api/routes/magento'));
app.listen(port);
logger.info('Magic happens on port ' + port);
gitextract__jw74e5m/
├── .gitignore
├── CHANGELOG.md
├── LICENSE
├── README.md
├── doc/
│ ├── destination_es_product.json
│ ├── product_mapping.json
│ ├── source_magento_category.json
│ ├── source_magento_product.json
│ └── source_magento_review.json
├── docker-compose.yml
├── package.json
└── src/
├── adapters/
│ ├── abstract.js
│ ├── factory.js
│ ├── magento/
│ │ ├── abstract.js
│ │ ├── attribute.js
│ │ ├── cache_keys.js
│ │ ├── category.js
│ │ ├── cms_block.js
│ │ ├── cms_page.js
│ │ ├── magento2-rest-client/
│ │ │ ├── .npmignore
│ │ │ ├── LICENSE
│ │ │ ├── README.md
│ │ │ ├── index.js
│ │ │ ├── lib/
│ │ │ │ ├── attributes.js
│ │ │ │ ├── blocks.js
│ │ │ │ ├── bundle_options.js
│ │ │ │ ├── categories.js
│ │ │ │ ├── category_products.js
│ │ │ │ ├── configurable_children.js
│ │ │ │ ├── configurable_options.js
│ │ │ │ ├── custom_options.js
│ │ │ │ ├── log.js
│ │ │ │ ├── pages.js
│ │ │ │ ├── product_links.js
│ │ │ │ ├── product_media.js
│ │ │ │ ├── products.js
│ │ │ │ ├── rest_client.js
│ │ │ │ ├── reviews.js
│ │ │ │ ├── stock_items.js
│ │ │ │ ├── tax_rates.js
│ │ │ │ └── tax_rules.js
│ │ │ ├── magento2-rest-client.iml
│ │ │ ├── package.json
│ │ │ └── test/
│ │ │ ├── config.json
│ │ │ └── integration/
│ │ │ ├── categories.integration.test.js
│ │ │ ├── product_media.integration.test.js
│ │ │ └── products.integration.test.js
│ │ ├── product.js
│ │ ├── productcategories.js
│ │ ├── review.js
│ │ └── taxrule.js
│ └── nosql/
│ ├── abstract.js
│ └── elasticsearch.js
├── api/
│ └── routes/
│ └── magento.js
├── cli.js
├── config.js
├── helpers/
│ └── slugify.js
├── log.js
├── test_by_sku.sh
├── test_categoryextended.sh
├── test_fullreindex.sh
├── test_fullreindex_de.sh
├── test_fullreindex_it.sh
├── test_fullreindex_multiprocess.sh
├── test_multistore.sh
├── test_product.sh
├── test_product_delta.sh
├── test_product_msi.sh
├── test_product_worker.sh
├── tmp/
│ └── .gitignore
└── webapi.js
SYMBOL INDEX (140 symbols across 17 files)
FILE: src/adapters/abstract.js
class AbstractAdapter (line 6) | class AbstractAdapter {
method validateConfig (line 8) | validateConfig(config) {
method constructor (line 13) | constructor(app_config) {
method isValidFor (line 47) | isValidFor(entity_type) {
method getCurrentContext (line 51) | getCurrentContext() {
method defaultDoneCallback (line 58) | defaultDoneCallback() {
method run (line 66) | run(context) {
method preProcessItem (line 97) | preProcessItem(item) {
method cleanUp (line 105) | cleanUp(transaction_key) {
method prepareItems (line 112) | prepareItems(items) {
method isFederated (line 125) | isFederated() {
method processItems (line 129) | processItems(items, level) {
FILE: src/adapters/factory.js
class AdapterFactory (line 3) | class AdapterFactory {
method constructor (line 5) | constructor (app_config) {
method getAdapter (line 9) | getAdapter (adapter_type, driver) {
FILE: src/adapters/magento/abstract.js
class AbstractMagentoAdapter (line 5) | class AbstractMagentoAdapter extends AbstractAdapter{
method constructor (line 7) | constructor(config){
method getEntityType (line 14) | getEntityType(){
method getCollectionName (line 18) | getCollectionName(){
method validateConfig (line 22) | validateConfig(config){
method isValidFor (line 34) | isValidFor(entity_type){
method getSourceData (line 38) | getSourceData(){
method getLabel (line 42) | getLabel(source_item){
method normalizeDocumentFormat (line 50) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/attribute.js
class AttributeAdapter (line 7) | class AttributeAdapter extends AbstractMagentoAdapter {
method getEntityType (line 9) | getEntityType() {
method getName (line 13) | getName() {
method getSourceData (line 17) | getSourceData(context) {
method prepareItems (line 22) | prepareItems(items) {
method getLabel (line 35) | getLabel(source_item) {
method isFederated (line 39) | isFederated() {
method preProcessItem (line 43) | preProcessItem(item) {
method normalizeDocumentFormat (line 65) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/category.js
class CategoryAdapter (line 28) | class CategoryAdapter extends AbstractMagentoAdapter {
method constructor (line 30) | constructor (config) {
method getEntityType (line 36) | getEntityType() {
method getName (line 40) | getName() {
method getSourceData (line 44) | getSourceData(context) {
method getLabel (line 50) | getLabel(source_item) {
method isFederated (line 54) | isFederated() {
method _addSingleCategoryData (line 58) | _addSingleCategoryData(item, result) {
method _extendSingleCategory (line 62) | _extendSingleCategory(rootId, catToExtend) {
method _extendChildrenCategories (line 73) | _extendChildrenCategories(rootId, children, result, subpromises) {
method preProcessItem (line 85) | preProcessItem(item) {
method normalizeDocumentFormat (line 143) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/cms_block.js
class BlockAdapter (line 5) | class BlockAdapter extends AbstractMagentoAdapter {
method constructor (line 6) | constructor(config) {
method getEntityType (line 11) | getEntityType() {
method getName (line 15) | getName() {
method getSourceData (line 19) | getSourceData(context) {
method prepareItems (line 31) | prepareItems(items) {
method isFederated (line 47) | isFederated() {
method preProcessItem (line 51) | preProcessItem(item) {
method normalizeDocumentFormat (line 65) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/cms_page.js
class PageAdapter (line 5) | class PageAdapter extends AbstractMagentoAdapter {
method constructor (line 6) | constructor(config) {
method getEntityType (line 11) | getEntityType() {
method getName (line 15) | getName() {
method getSourceData (line 19) | getSourceData(context) {
method prepareItems (line 31) | prepareItems(items) {
method preProcessItem (line 45) | preProcessItem(item) {
method normalizeDocumentFormat (line 57) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/magento2-rest-client/index.js
constant MAGENTO_API_VERSION (line 21) | const MAGENTO_API_VERSION = 'V1';
FILE: src/adapters/magento/magento2-rest-client/lib/rest_client.js
function apiCall (line 27) | function apiCall(request_data) {
function httpCallSucceeded (line 57) | function httpCallSucceeded(response) {
function errorString (line 61) | function errorString(message, parameters) {
function createUrl (line 88) | function createUrl(resourceUrl) {
FILE: src/adapters/magento/product.js
constant HTTP_RETRIES (line 9) | const HTTP_RETRIES = 3
class ProductAdapter (line 38) | class ProductAdapter extends AbstractMagentoAdapter {
method constructor (line 40) | constructor(config) {
method getEntityType (line 54) | getEntityType() {
method getName (line 58) | getName() {
method prepareItems (line 62) | prepareItems(items) {
method getFilterQuery (line 76) | getFilterQuery(context) {
method getSourceData (line 95) | getSourceData(context) {
method getProductSourceData (line 163) | getProductSourceData(context) {
method getTotalCount (line 218) | getTotalCount(context) {
method getLabel (line 223) | getLabel(source_item) {
method isNumeric (line 227) | isNumeric(value) {
method processAttributes (line 231) | processAttributes(customAttributes, configurableOptions) {
method preProcessItem (line 301) | preProcessItem(item) {
method computeVideoData (line 668) | computeVideoData(mediaItem) {
method normalizeDocumentFormat (line 704) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/productcategories.js
class ProductcategoriesAdapter (line 20) | class ProductcategoriesAdapter extends AbstractMagentoAdapter {
method constructor (line 22) | constructor(config) {
method getEntityType (line 32) | getEntityType() {
method getName (line 36) | getName() {
method getSourceData (line 40) | getSourceData(context) {
method getLabel (line 44) | getLabel(source_item) {
method _storeCatLinks (line 52) | _storeCatLinks(item, result) {
method preProcessItem (line 97) | preProcessItem(item) {
method defaultDoneCallback (line 114) | defaultDoneCallback() {
method isFederated (line 133) | isFederated() {
method normalizeDocumentFormat (line 141) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/review.js
class ReviewAdapter (line 5) | class ReviewAdapter extends AbstractMagentoAdapter {
method constructor (line 6) | constructor(config) {
method getEntityType (line 11) | getEntityType() {
method getName (line 15) | getName() {
method getSourceData (line 19) | getSourceData(context) {
method prepareItems (line 35) | prepareItems(items) {
method isFederated (line 55) | isFederated() {
method preProcessItem (line 59) | preProcessItem(item) {
method normalizeDocumentFormat (line 79) | normalizeDocumentFormat(item) {
FILE: src/adapters/magento/taxrule.js
class TaxruleAdapter (line 5) | class TaxruleAdapter extends AbstractMagentoAdapter {
method getEntityType (line 7) | getEntityType() {
method getName (line 11) | getName() {
method getSourceData (line 15) | getSourceData(context) {
method getLabel (line 19) | getLabel(source_item) {
method isFederated (line 23) | isFederated() {
method preProcessItem (line 27) | preProcessItem(item) {
method prepareItems (line 53) | prepareItems(items) {
method normalizeDocumentFormat (line 65) | normalizeDocumentFormat(item) {
FILE: src/adapters/nosql/abstract.js
class AbstractNosqlAdapter (line 4) | class AbstractNosqlAdapter{
method validateConfig (line 6) | validateConfig(config){
method constructor (line 13) | constructor(app_config){
method getWhereQuery (line 24) | getWhereQuery(source_item){
method close (line 31) | close(){
method updateDocument (line 39) | updateDocument(collectionName, item) {
method getDocuments (line 48) | getDocuments(collectionName, query) {
method updateDocumentBulk (line 55) | updateDocumentBulk(collectionName, items){
method cleanupByTransactionkey (line 64) | cleanupByTransactionkey(collectionName, transactionKey){
method connect (line 71) | connect (done){
FILE: src/adapters/nosql/elasticsearch.js
class ElasticsearchAdapter (line 8) | class ElasticsearchAdapter extends AbstractNosqlAdapter {
method validateConfig (line 10) | validateConfig(config) {
method constructor (line 20) | constructor(app_config) {
method getPhysicalIndexName (line 36) | getPhysicalIndexName(collectionName, config) {
method getPhysicalTypeName (line 49) | getPhysicalTypeName(collectionName, config) {
method close (line 60) | close() { // switched to global singleton
method getDocuments (line 69) | getDocuments(collectionName, queryBody) {
method updateDocument (line 93) | updateDocument(collectionName, item) {
method cleanupByTransactionkey (line 119) | cleanupByTransactionkey(collectionName, transactionKey) {
method updateDocumentBulk (line 148) | updateDocumentBulk(collectionName, items) {
method connect (line 195) | connect(done) {
FILE: src/cli.js
constant TIME_TO_EXIT (line 8) | const TIME_TO_EXIT = process.env.TIME_TO_EXIT ? process.env.TIME_TO_EXIT...
constant INDEX_META_PATH (line 14) | const INDEX_META_PATH = process.env.INDEX_META_PATH ? path.join('tmp', p...
function cleanup (line 179) | function cleanup(adapterName, cleanupType, transactionKey) {
function reindexProducts (line 192) | function reindexProducts(adapterName, removeNonExistent, partitions, par...
function fullReindex (line 293) | function fullReindex(adapterName, removeNonExistent, partitions, partiti...
function runProductsworker (line 312) | function runProductsworker(adapterName, partitions) {
FILE: src/helpers/slugify.js
function slugify (line 8) | function slugify (text) {
Condensed preview — 71 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (175K chars).
[
{
"path": ".gitignore",
"chars": 94,
"preview": "src/test_cleanup.sh\nsrc/test_reindex.sh\nsrc/node_modules\npackage-lock.json\nnode_modules\n.idea\n"
},
{
"path": "CHANGELOG.md",
"chars": 2563,
"preview": "# Changelog\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Changel"
},
{
"path": "LICENSE",
"chars": 1069,
"preview": "MIT License\n\nCopyright (c) 2017 Divante Ltd.\n\nPermission is hereby granted, free of charge, to any person obtaining a co"
},
{
"path": "README.md",
"chars": 20051,
"preview": "# mage2alokai\n\n### Stay connected\n\n[![GitHub Repo stars](https://img.shields.io/github/stars/vuestorefront/vue-storefron"
},
{
"path": "doc/destination_es_product.json",
"chars": 4479,
"preview": "\"entity_id\": \"1351\",\n\"attribute_set_id\": \"10\",\n\"type_id\": \"simple\",\n\"sku\": \"6343908\",\n\"has_options\": \"0\",\n\"required_opti"
},
{
"path": "doc/product_mapping.json",
"chars": 5202,
"preview": "{\n \"vue_storefront_catalog\": {\n \"mappings\": {\n \"product\": {\n \"properties\": {\n "
},
{
"path": "doc/source_magento_category.json",
"chars": 180,
"preview": "{\n \"id\": 0,\n \"parent_id\": 0,\n \"name\": \"string\",\n \"is_active\": true,\n \"position\": 0,\n \"level\": 0,\n \""
},
{
"path": "doc/source_magento_product.json",
"chars": 6516,
"preview": "\nModelModel Schema\n{\n \"items\": [\n {\n \"id\": 0,\n \"sku\": \"string\",\n \"name\": \"string\",\n \"attribute_s"
},
{
"path": "doc/source_magento_review.json",
"chars": 343,
"preview": "{\n \"id\": 0,\n \"title\": \"string\",\n \"detail\": \"string\",\n \"nickname\": \"string\",\n \"ratings\": [{\n \"id\": 0,\n \"review"
},
{
"path": "docker-compose.yml",
"chars": 965,
"preview": "version: '2'\nservices:\n esm1:\n image: elasticsearch:7.3.2\n container_name: esm1\n environment:\n - cluster."
},
{
"path": "package.json",
"chars": 1012,
"preview": "{\n \"name\": \"mage2vuestorefront\",\n \"private\": true,\n \"version\": \"1.11.1\",\n \"description\": \"Magento sync for products,"
},
{
"path": "src/adapters/abstract.js",
"chars": 6389,
"preview": "'use strict';\n\nconst AdapterFactory = require('./factory');\nconst Redis = require('redis');\n\nclass AbstractAdapter {\n\n "
},
{
"path": "src/adapters/factory.js",
"chars": 653,
"preview": "'use strict';\n\nclass AdapterFactory {\n\n constructor (app_config) {\n this.config = app_config;\n }\n\n getAdapter (ad"
},
{
"path": "src/adapters/magento/abstract.js",
"chars": 1357,
"preview": "'use strict';\n\nlet AbstractAdapter = require('../abstract');\n\nclass AbstractMagentoAdapter extends AbstractAdapter{\n\n c"
},
{
"path": "src/adapters/magento/attribute.js",
"chars": 1916,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\nconst CacheKeys = require('./cache_keys');\nconst util"
},
{
"path": "src/adapters/magento/cache_keys.js",
"chars": 433,
"preview": "var config = require('../../config')\n\nmodule.exports = {\n CACHE_KEY_CATEGORY: config.db.indexName + '_cat_%s',\n CACHE_"
},
{
"path": "src/adapters/magento/category.js",
"chars": 4744,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\nconst CacheKeys = require('./cache_keys');\nconst util"
},
{
"path": "src/adapters/magento/cms_block.js",
"chars": 1657,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\n\nclass BlockAdapter extends AbstractMagentoAdapter {\n"
},
{
"path": "src/adapters/magento/cms_page.js",
"chars": 1362,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\n\nclass PageAdapter extends AbstractMagentoAdapter {\n "
},
{
"path": "src/adapters/magento/magento2-rest-client/.npmignore",
"chars": 529,
"preview": "# Logs\nlogs\n*.log\nnpm-debug.log*\n\n# Runtime data\npids\n*.pid\n*.seed\n\n# Directory for instrumented libs generated by jscov"
},
{
"path": "src/adapters/magento/magento2-rest-client/LICENSE",
"chars": 1078,
"preview": "The MIT License (MIT)\n\nCopyright (c) 2016 Marko Novak\n\nPermission is hereby granted, free of charge, to any person obtai"
},
{
"path": "src/adapters/magento/magento2-rest-client/README.md",
"chars": 1001,
"preview": "# Magento2 REST client\n\nThis Node.js library enables JavaScript applications to communicate with Magento2 sites using th"
},
{
"path": "src/adapters/magento/magento2-rest-client/index.js",
"chars": 1843,
"preview": "'use strict';\n\nvar RestClient = require('./lib/rest_client').RestClient;\nvar categories = require('./lib/categories');\nv"
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/attributes.js",
"chars": 871,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/blocks.js",
"chars": 342,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function(s"
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/bundle_options.js",
"chars": 299,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/categories.js",
"chars": 870,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/category_products.js",
"chars": 285,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/configurable_children.js",
"chars": 302,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/configurable_options.js",
"chars": 305,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/custom_options.js",
"chars": 288,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/log.js",
"chars": 395,
"preview": "var winston = require('winston');\n\nwinston.emitErrs = true;\n\nvar logger = new winston.Logger({\n transports: [\n "
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/pages.js",
"chars": 342,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/product_links.js",
"chars": 715,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n var typesCache = null;\n\n "
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/product_media.js",
"chars": 1211,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/products.js",
"chars": 1197,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/rest_client.js",
"chars": 3729,
"preview": "'use strict';\n\nvar OAuth = require('oauth-1.0a');\nvar request = require('request');\nvar humps = require('humps');\nvar sp"
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/reviews.js",
"chars": 798,
"preview": "const util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.getByProductSku"
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/stock_items.js",
"chars": 873,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/tax_rates.js",
"chars": 728,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/lib/tax_rules.js",
"chars": 798,
"preview": "var util = require('util');\n\nmodule.exports = function (restClient) {\n var module = {};\n\n module.list = function ("
},
{
"path": "src/adapters/magento/magento2-rest-client/magento2-rest-client.iml",
"chars": 426,
"preview": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<module type=\"WEB_MODULE\" version=\"4\">\n <component name=\"NewModuleRootManager\" i"
},
{
"path": "src/adapters/magento/magento2-rest-client/package.json",
"chars": 1684,
"preview": "{\n \"_from\": \"magento2-rest-client@0.0.2\",\n \"_id\": \"magento2-rest-client@0.0.2\",\n \"_inBundle\": false,\n \"_integrity\": "
},
{
"path": "src/adapters/magento/magento2-rest-client/test/config.json",
"chars": 272,
"preview": "{\n \"url\": \"http://www.igracke.si/index.php/rest\",\n \"consumerKey\": \"h0vun4y13ov0pg1drcndfhnim761omcu\",\n \"consumerSecre"
},
{
"path": "src/adapters/magento/magento2-rest-client/test/integration/categories.integration.test.js",
"chars": 1697,
"preview": "var chai = require('chai');\nvar credentials = require('../config');\nvar assert = chai.assert;\n\nvar Magento2Client = requ"
},
{
"path": "src/adapters/magento/magento2-rest-client/test/integration/product_media.integration.test.js",
"chars": 2910,
"preview": "var chai = require('chai');\nvar credentials = require('../config');\nvar assert = chai.assert;\n\nvar Magento2Client = requ"
},
{
"path": "src/adapters/magento/magento2-rest-client/test/integration/products.integration.test.js",
"chars": 1948,
"preview": "var chai = require('chai');\nvar credentials = require('../config');\nvar assert = chai.assert;\n\nvar Magento2Client = requ"
},
{
"path": "src/adapters/magento/product.js",
"chars": 27770,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\nconst util = require('util');\nconst CacheKeys = requi"
},
{
"path": "src/adapters/magento/productcategories.js",
"chars": 4149,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\n\nlet queue = require('queue');\nlet util = require('ut"
},
{
"path": "src/adapters/magento/review.js",
"chars": 1842,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\n\nclass ReviewAdapter extends AbstractMagentoAdapter {"
},
{
"path": "src/adapters/magento/taxrule.js",
"chars": 1536,
"preview": "'use strict';\n\nlet AbstractMagentoAdapter = require('./abstract');\n\nclass TaxruleAdapter extends AbstractMagentoAdapter "
},
{
"path": "src/adapters/nosql/abstract.js",
"chars": 1743,
"preview": "'use strict';\n\n\nclass AbstractNosqlAdapter{\n\n validateConfig(config){\n\n if(!config['db']['url'])\n throw Error('"
},
{
"path": "src/adapters/nosql/elasticsearch.js",
"chars": 5935,
"preview": "'use strict';\nconst AbstractNosqlAdapter = require('./abstract');\nconst elasticsearch = require('@elastic/elasticsearch'"
},
{
"path": "src/api/routes/magento.js",
"chars": 851,
"preview": "'use strict';\nlet express = require('express');\nlet kue = require('kue');\n\nlet config = require('../../config');\nlet Ada"
},
{
"path": "src/cli.js",
"chars": 17161,
"preview": "'use strict';\n\nconst program = require('commander');\nconst fs = require('fs');\nconst path = require('path');\nlet Adapter"
},
{
"path": "src/config.js",
"chars": 3066,
"preview": "const _slugify = require('./helpers/slugify')\n\nmodule.exports = {\n\n seo: {\n useUrlDispatcher: JSON.parse(process.env"
},
{
"path": "src/helpers/slugify.js",
"chars": 600,
"preview": "const removeAccents = require('remove-accents').remove\n\n/**\n * Create slugify -> \"create-slugify\" permalink of text\n * "
},
{
"path": "src/log.js",
"chars": 411,
"preview": "var winston = require('winston');\n\nwinston.emitErrs = true;\n\nif(!global.logger) {\n global.logger = new winston.Logger({"
},
{
"path": "src/test_by_sku.sh",
"chars": 613,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_categoryextended.sh",
"chars": 593,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_fullreindex.sh",
"chars": 973,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_fullreindex_de.sh",
"chars": 994,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_fullreindex_it.sh",
"chars": 994,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_fullreindex_multiprocess.sh",
"chars": 916,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_multistore.sh",
"chars": 2082,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_product.sh",
"chars": 616,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_product_delta.sh",
"chars": 582,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_product_msi.sh",
"chars": 678,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/test_product_worker.sh",
"chars": 583,
"preview": "\nexport TIME_TO_EXIT=2000\nexport VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=\nexport VS_IN"
},
{
"path": "src/tmp/.gitignore",
"chars": 8,
"preview": "*.json\n\n"
},
{
"path": "src/webapi.js",
"chars": 578,
"preview": "'use strict';\n\n/**\n * Webhook API to add specific products or categories to be synchronized by the service\n */\n\nlet logg"
}
]
About this extraction
This page contains the full source code of the DivanteLtd/mage2vuestorefront GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 71 files (160.2 KB), approximately 41.4k tokens, and a symbol index with 140 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.