[
{
"path": ".aws/task_definition.json",
"content": "{\n \"taskDefinitionArn\": \"arn:aws:ecs:eu-west-1:253053805092:task-definition/megaparse-task:2\",\n \"containerDefinitions\": [\n {\n \"name\": \"megaparse\",\n \"image\": \"quay.io/unstructured-io/unstructured-api:latest\",\n \"cpu\": 0,\n \"portMappings\": [\n {\n \"containerPort\": 8000,\n \"hostPort\": 8000,\n \"protocol\": \"tcp\"\n }\n ],\n \"essential\": true,\n \"environment\": [\n {\n \"name\": \"UNSTRUCTURED_HI_RES_MODEL_NAME\",\n \"value\": \"detectron2_onnx\"\n },\n {\n \"name\": \"UNSTRUCTURED_PARALLEL_MODE_ENABLED\",\n \"value\": \"false\"\n }\n ],\n \"mountPoints\": [],\n \"volumesFrom\": [],\n \"logConfiguration\": {\n \"logDriver\": \"awslogs\",\n \"options\": {\n \"awslogs-group\": \"/ecs/megaparse\",\n \"awslogs-region\": \"eu-west-1\",\n \"awslogs-stream-prefix\": \"ecs\"\n }\n },\n \"systemControls\": []\n }\n ],\n \"family\": \"megaparse-task\",\n \"executionRoleArn\": \"arn:aws:iam::253053805092:role/megaparse-ecsTaskExecutionRole\",\n \"networkMode\": \"awsvpc\",\n \"revision\": 2,\n \"volumes\": [],\n \"status\": \"ACTIVE\",\n \"requiresAttributes\": [\n {\n \"name\": \"com.amazonaws.ecs.capability.logging-driver.awslogs\"\n },\n {\n \"name\": \"ecs.capability.execution-role-awslogs\"\n },\n {\n \"name\": \"com.amazonaws.ecs.capability.docker-remote-api.1.19\"\n },\n {\n \"name\": \"com.amazonaws.ecs.capability.docker-remote-api.1.18\"\n },\n {\n \"name\": \"ecs.capability.task-eni\"\n }\n ],\n \"placementConstraints\": [],\n \"compatibilities\": [\n \"EC2\",\n \"FARGATE\"\n ],\n \"requiresCompatibilities\": [\n \"FARGATE\"\n ],\n \"cpu\": \"2048\",\n \"memory\": \"8192\",\n \"tags\": []\n}"
},
{
"path": ".flake8",
"content": "[flake8]\n; Minimal configuration for Flake8 to work with Black.\nmax-line-length = 100\nignore = E101,E111,E112,E221,E222,E501,E711,E712,W503,W504,F401,BLK100\n"
},
{
"path": ".gitattributes",
"content": "*.ipynb linguist-vendored\n*.html linguist-vendored"
},
{
"path": ".github/workflows/CI.yml",
"content": "name: Run tests\n\non:\n pull_request:\n workflow_dispatch:\n\nenv:\n NATS_TOKEN: test\n\njobs:\n test:\n name: Run tests on Python ${{ matrix.python-version }}\n runs-on: ubuntu-latest\n strategy:\n matrix:\n python-version: [\"3.11\", \"3.12\"]\n steps:\n - name: π Checkout code\n uses: actions/checkout@v2\n with:\n submodules: true\n\n - name: Setup apt cache\n uses: actions/cache@v2\n with:\n path: /var/cache/apt/archives\n key: ${{ runner.os }}-apt-${{ hashFiles('/etc/apt/sources.list') }}\n\n - name: π Install system dependencies\n run: |\n sudo apt-get update && sudo apt-get install -y \\\n netcat-traditional \\\n unzip \\\n libgeos-dev \\\n libcurl4-openssl-dev \\\n libssl-dev \\\n binutils \\\n curl \\\n git \\\n autoconf \\\n automake \\\n build-essential \\\n libtool \\\n gcc \\\n libmagic-dev \\\n poppler-utils \\\n tesseract-ocr \\\n libreoffice \\\n libpq-dev \\\n pandoc\n\n - name: π½ Install the latest version of rye\n uses: eifinger/setup-rye@v4\n with:\n enable-cache: true\n\n - name: π Pin Python version\n run: rye pin ${{ matrix.python-version }}\n\n - name: π½ Download and Install NATS Server\n run: |\n curl -L https://github.com/nats-io/nats-server/releases/download/v2.10.22/nats-server-v2.10.22-linux-amd64.zip -o nats-server.zip\n unzip nats-server.zip -d nats-server && sudo cp nats-server/nats-server-v2.10.22-linux-amd64/nats-server /usr/bin\n\n - name: π οΈ Set up NATS arguments\n run: |\n nohup nats-server \\\n --addr 0.0.0.0 \\\n --port 4222 \\\n --auth \"$NATS_TOKEN\" > nats.log 2>&1 &\n\n - name: π Verify NATS Server is Running\n run: |\n sleep 1 # Give the server some time to start\n if nc -zv localhost 4222; then\n echo \"β NATS Server is running on port 4222.\"\n else\n echo \"β Failed to start NATS Server.\"\n cat nats.log\n exit 1\n fi\n\n - name: π¨ Sync dependencies\n run: |\n UV_INDEX_STRATEGY=unsafe-first-match rye sync --no-lock\n\n - name: π Run tests\n run: |\n rye test -p megaparse-sdk\n"
},
{
"path": ".github/workflows/build-and-deploy.yml",
"content": "name: Build Docker image and push ECR\n\non:\n push:\n tags:\n - \"v*\"\n branches: [main]\n\nenv:\n AWS_REGION: eu-west-1\n ECR_REPOSITORY: quivrhq/megaparse\n ECS_CLUSTER: megaparse\n ECS_TASK_DEFINITION: .aws/task_definition.json\n CONTAINER_NAME: megaparse\n\npermissions:\n contents: read\n\njobs:\n deploy:\n name: build docker\n runs-on: ubuntu-latest\n environment: production\n outputs:\n imageoutput: ${{ steps.build-image.outputs.imageoutput }}\n\n steps:\n - name: Checkout\n uses: actions/checkout@v3\n\n - name: Configure AWS credentials\n uses: aws-actions/configure-aws-credentials@v4\n with:\n aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}\n aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n aws-region: us-east-1\n\n - name: Login to Amazon ECR\n id: login-ecr\n uses: aws-actions/amazon-ecr-login@v1\n with:\n registry-type: public\n\n - name: Build, tag, and push image to Amazon ECR\n id: build-image\n env:\n ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}\n IMAGE_TAG: ${{ github.sha }}\n run: |\n # Build a docker container and push it to ECR\n docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .\n docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG\n\n # Tag the image as 'latest' and push\n docker tag $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG $ECR_REGISTRY/$ECR_REPOSITORY:latest\n docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest\n\n echo \"imageoutput=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG\" >> $GITHUB_OUTPUT\n"
},
{
"path": ".github/workflows/build-gpu.yml",
"content": "name: Build docker GPU and push ECR\n\non:\n push:\n tags:\n - \"v*\"\n branches: [main]\n\nenv:\n AWS_REGION: eu-west-1\n ECR_REPOSITORY: quivrhq/megaparse-gpu\n ECS_CLUSTER: megaparse\n ECS_TASK_DEFINITION: .aws/task_definition.json\n CONTAINER_NAME: megaparse\n\npermissions:\n contents: read\n\njobs:\n deploy:\n name: Build docker-gpu\n runs-on:\n group: big-boy-gpu\n environment: production\n outputs:\n imageoutput: ${{ steps.build-image.outputs.imageoutput }}\n\n steps:\n - name: Checkout\n uses: actions/checkout@v3\n\n - name: Configure AWS credentials\n uses: aws-actions/configure-aws-credentials@v4\n with:\n aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}\n aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n aws-region: us-east-1\n\n - name: Login to Amazon ECR\n id: login-ecr\n uses: aws-actions/amazon-ecr-login@v1\n with:\n registry-type: public\n\n - name: Build, tag, and push image to Amazon ECR\n id: build-image\n env:\n ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}\n IMAGE_TAG: ${{ github.sha }}\n run: |\n # Build a docker container and push it to ECR\n docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG -f Dockerfile.gpu .\n docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG\n\n # Tag the image as 'latest' and push\n docker tag $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG $ECR_REGISTRY/$ECR_REPOSITORY:latest\n docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest\n\n echo \"imageoutput=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG\" >> $GITHUB_OUTPUT\n"
},
{
"path": ".github/workflows/release-please.yml",
"content": "on:\n push:\n branches:\n - main\n\npermissions:\n contents: write\n pull-requests: write\n\nname: release-please\n\njobs:\n release-please:\n runs-on: ubuntu-latest\n outputs:\n release_created: ${{ steps.release.outputs['libs/megaparse--release_created'] }}\n release_created_sdk: ${{ steps.release.outputs['libs/megaparse_sdk--release_created'] }}\n steps:\n - name: Checkout repository\n uses: actions/checkout@v3\n with:\n fetch-depth: 0 # Fetch all history for tags and releases\n\n - name: Setup Python\n uses: actions/setup-python@v4\n with:\n python-version: \"3.11\"\n\n - name: Run release-please\n id: release\n uses: google-github-actions/release-please-action@v4\n with:\n token: ${{ secrets.RELEASE_PLEASE_TOKEN }}\n\n deploy-megaparse:\n if: needs.release-please.outputs.release_created == 'true'\n needs: release-please\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - name: Install Rye\n uses: eifinger/setup-rye@v2\n with:\n enable-cache: true\n - name: Rye Sync\n run: rye sync --no-lock\n - name: Rye Build\n run: cd libs/megaparse && rye build\n - name: Rye Publish\n run: cd libs/megaparse && rye publish --token ${{ secrets.PYPI_API_TOKEN }} --yes\n\n deploy-sdk:\n if: needs.release-please.outputs.release_created_sdk == 'true'\n needs: release-please\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - name: Install Rye\n uses: eifinger/setup-rye@v2\n with:\n enable-cache: true\n - name: Rye Sync\n run: cd libs/megaparse_sdk && rye sync --no-lock\n - name: Rye Build\n run: cd libs/megaparse_sdk && rye build\n - name: Rye Publish\n run: cd libs/megaparse_sdk && rye publish --token ${{ secrets.PYPI_API_TOKEN }} --yes\n"
},
{
"path": ".github/workflows/test-build-docker.yml",
"content": "on:\n pull_request:\n branches:\n - main\n\nname: Test build docker\njobs:\n build-docker:\n runs-on: ubuntu-latest\n strategy:\n matrix:\n dockerfile: [Dockerfile, Dockerfile.gpu]\n steps:\n - name: Checkout repository\n uses: actions/checkout@v3\n\n - name: Set up QEMU\n uses: docker/setup-qemu-action@v3\n with:\n platforms: all\n\n - name: Set up Docker Buildx\n uses: docker/setup-buildx-action@v3\n\n - name: Build Docker image with caching\n uses: docker/build-push-action@v4\n with:\n context: .\n file: ${{ matrix.dockerfile }}\n push: false\n tags: quivrhq/megaparse:${{ matrix.dockerfile }}\n cache-from: type=gha\n cache-to: type=gha,mode=max\n"
},
{
"path": ".gitignore",
"content": "/output\n/input\n.env\n__pycache__/\ndist/**\nmegaparse.egg-info/\n*.pyc\nbuild/*\nENV\nvenv\n*/evaluations/*\n*/cdp/*\n*.pkl\n\n!megaparse/tests/output_tests/MegaFake_report.md\n*.DS_Store\n.tool-versions\nmegaparse/sdk/examples/only_pdfs/*\n\n**/profile/\n**/prof/\n.ropeproject/\nbenchmark/hi_res/*\nbenchmark/auto/*\n\n"
},
{
"path": ".pre-commit-config.yaml",
"content": "repos:\n - repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v4.6.0\n hooks:\n - id: check-added-large-files\n args: [\"--maxkb=5000\"]\n - id: check-toml\n - id: check-yaml\n - id: end-of-file-fixer\n - id: trailing-whitespace\n - id: check-merge-conflict\n - id: detect-private-key\n - id: check-case-conflict\n - repo: https://github.com/pre-commit/pre-commit\n rev: v3.6.2\n hooks:\n - id: validate_manifest\n - repo: https://github.com/astral-sh/ruff-pre-commit\n # Ruff version.\n rev: v0.5.1\n hooks:\n # Run the linter.\n - id: ruff\n args: [--fix]\n additional_dependencies: []\n # Run the formatter.\n - id: ruff-format\n additional_dependencies: []\n - repo: https://github.com/pre-commit/mirrors-mypy\n rev: v1.10.1\n hooks:\n - id: mypy\n name: mypy\n additional_dependencies: [\"types-aiofiles\"]\n"
},
{
"path": ".python-version",
"content": "3.11.9\n"
},
{
"path": ".release-please-manifest.json",
"content": "{\n \"libs/megaparse\": \"0.0.55\",\n \"libs/megaparse_sdk\": \"0.1.12\"\n}\n"
},
{
"path": ".vscode/extensions.json",
"content": "{\n \"recommendations\": [\n \"dbaeumer.vscode-eslint\",\n \"charliermarsh.ruff\",\n \"knisterpeter.vscode-github\",\n \"github.vscode-pull-request-github\",\n \"ms-python.python\",\n \"ms-python.vscode-pylance\",\n \"ms-python.debugpy\"\n ]\n}\n"
},
{
"path": ".vscode/launch.json",
"content": "{\n \"version\": \"0.2.0\",\n \"configurations\": [\n {\n \"name\": \"Python: Remote Attach\",\n \"type\": \"python\",\n \"request\": \"attach\",\n \"connect\": {\n \"host\": \"localhost\",\n \"port\": 5678\n },\n \"pathMappings\": [\n {\n \"localRoot\": \"${workspaceFolder}/backend\",\n \"remoteRoot\": \".\"\n }\n ],\n \"justMyCode\": true\n },\n {\n \"name\": \"Python: Debug Test Script\",\n \"type\": \"python\",\n \"request\": \"launch\",\n \"program\": \"${workspaceFolder}/backend/test_process_file_and_notify.py\",\n \"console\": \"integratedTerminal\",\n \"justMyCode\": false\n },\n {\n \"name\": \"Python: Debug\",\n \"type\": \"debugpy\",\n \"request\": \"launch\",\n \"program\": \"${file}\",\n \"console\": \"integratedTerminal\",\n \"justMyCode\": false,\n \"env\": {\n \"PYTHONPATH\": \"${workspaceFolder}/backend:${env:PYTHONPATH}\"\n },\n \"envFile\": \"${workspaceFolder}/.env\"\n }\n ]\n}\n"
},
{
"path": ".vscode/settings.json",
"content": "{\n \"editor.formatOnSave\": true,\n \"editor.formatOnSaveMode\": \"file\",\n \"files.exclude\": {\n \"**/__pycache__\": true,\n \"**/.benchmarks/\": true,\n \"**/.cache/\": true,\n \"**/.pytest_cache/\": true,\n \"**/.next/\": true,\n \"**/build/\": true,\n \"**/.docusaurus/\": true,\n \"**/node_modules/\": true\n },\n \"[python]\": {\n \"editor.defaultFormatter\": \"charliermarsh.ruff\",\n \"editor.formatOnSave\": true,\n \"editor.codeActionsOnSave\": {\n \"source.organizeImports\": \"explicit\",\n \"source.fixAll\": \"explicit\"\n }\n },\n \"python.testing.unittestEnabled\": false,\n \"python.testing.pytestEnabled\": true,\n \"python.testing.autoTestDiscoverOnSaveEnabled\": true,\n \"python.analysis.autoImportCompletions\": true,\n \"python.analysis.typeCheckingMode\": \"basic\",\n \"python.analysis.diagnosticSeverityOverrides\": {\n \"reportMissingImports\": \"error\",\n \"reportUnusedImport\": \"warning\",\n \"reportGeneralTypeIssues\": \"warning\"\n },\n \"makefile.configureOnOpen\": false\n}\n"
},
{
"path": "CHANGELOG.md",
"content": "# Changelog\n\n## [0.0.46](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.45...megaparse-v0.0.46) (2024-11-21)\n\n\n### Features\n\n* refacto megaparse for service ([#132](https://github.com/QuivrHQ/MegaParse/issues/132)) ([ab9ad7f](https://github.com/QuivrHQ/MegaParse/commit/ab9ad7fb7db580a04a998d144dd2ba3407068334))\n\n## [0.0.45](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.44...megaparse-v0.0.45) (2024-11-19)\n\n\n### Bug Fixes\n\n* small fixes from backlogs ([#128](https://github.com/QuivrHQ/MegaParse/issues/128)) ([954554c](https://github.com/QuivrHQ/MegaParse/commit/954554c5abaa7b0513e9ff3f6bbaff393d36cf03))\n\n## [0.0.44](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.43...megaparse-v0.0.44) (2024-11-18)\n\n\n### Bug Fixes\n\n* fixing the wrong passing of arguments to the parse_file endpoint ([#123](https://github.com/QuivrHQ/MegaParse/issues/123)) ([9105672](https://github.com/QuivrHQ/MegaParse/commit/9105672abc0942f26785e494053112d486e8d2d9))\n\n## [0.0.43](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.42...megaparse-v0.0.43) (2024-11-14)\n\n\n### Features\n\n* increase the robustness of megaparse ([#121](https://github.com/QuivrHQ/MegaParse/issues/121)) ([d21d8bb](https://github.com/QuivrHQ/MegaParse/commit/d21d8bb77bd8e687b1a951db6b81653e4e47a8bb))\n\n\n### Bug Fixes\n\n* uvicorn version ([#127](https://github.com/QuivrHQ/MegaParse/issues/127)) ([ceaba3d](https://github.com/QuivrHQ/MegaParse/commit/ceaba3df2951be27e6a4835e5784917a62867896))\n* version requirements ([#126](https://github.com/QuivrHQ/MegaParse/issues/126)) ([a10d502](https://github.com/QuivrHQ/MegaParse/commit/a10d502f1b3576690cebe33b656d2480a24defe3))\n\n## [0.0.42](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.41...megaparse-v0.0.42) (2024-11-08)\n\n\n### Features\n\n* **sdk:** new version ([e377cd6](https://github.com/QuivrHQ/MegaParse/commit/e377cd6df98b3ea9265788a4d907b43bde796196))\n\n## [0.0.41](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.40...megaparse-v0.0.41) (2024-11-08)\n\n\n### Bug Fixes\n\n* add megaparse url env variable ([#118](https://github.com/QuivrHQ/MegaParse/issues/118)) ([132c2eb](https://github.com/QuivrHQ/MegaParse/commit/132c2ebd13177fd116c4e710a4b1c864a9fa04bb))\n\n## [0.0.40](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.39...megaparse-v0.0.40) (2024-11-08)\n\n\n### Bug Fixes\n\n* sdk version ([#116](https://github.com/QuivrHQ/MegaParse/issues/116)) ([8bfeb4a](https://github.com/QuivrHQ/MegaParse/commit/8bfeb4a52326a5f645d3ed20e113153dc19bf012))\n\n## [0.0.39](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.38...megaparse-v0.0.39) (2024-11-08)\n\n\n### Bug Fixes\n\n* add_logs ([#114](https://github.com/QuivrHQ/MegaParse/issues/114)) ([63c9236](https://github.com/QuivrHQ/MegaParse/commit/63c9236590016ee4c210174e746e96ff2b654480))\n\n## [0.0.38](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.37...megaparse-v0.0.38) (2024-11-07)\n\n\n### Bug Fixes\n\n* env roots, imports root ([#112](https://github.com/QuivrHQ/MegaParse/issues/112)) ([a04230d](https://github.com/QuivrHQ/MegaParse/commit/a04230dc2de9e0bb0bde39ab66b2208f80743922))\n\n## [0.0.37](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.36...megaparse-v0.0.37) (2024-11-07)\n\n\n### Features\n\n* bump megaparse-sdk version to 0.1.1 ([ed3fdfb](https://github.com/QuivrHQ/MegaParse/commit/ed3fdfb10498c95d4f9a510df3a2913e0dfc3c23))\n\n## [0.0.36](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.35...megaparse-v0.0.36) (2024-11-07)\n\n\n### Features\n\n* **readme:** update ([9d571b7](https://github.com/QuivrHQ/MegaParse/commit/9d571b7c71db610e7a0b08045ad98994ecf71baa))\n\n## [0.0.35](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.34...megaparse-v0.0.35) (2024-11-07)\n\n\n### Bug Fixes\n\n* unnecessary dep and readme ([#107](https://github.com/QuivrHQ/MegaParse/issues/107)) ([b80aaa3](https://github.com/QuivrHQ/MegaParse/commit/b80aaa3a894b2bd2c7d7f518919c41af5c99219f))\n\n## [0.0.34](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.33...megaparse-v0.0.34) (2024-11-07)\n\n\n### Features\n\n* megaparse-sdk-cherry ([#105](https://github.com/QuivrHQ/MegaParse/issues/105)) ([ad44aa3](https://github.com/QuivrHQ/MegaParse/commit/ad44aa34999596e156c78f91adab97bce7ceeb0e))\n\n## [0.0.33](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.32...megaparse-v0.0.33) (2024-11-01)\n\n\n### Bug Fixes\n\n* readme ([#99](https://github.com/QuivrHQ/MegaParse/issues/99)) ([b3b80a3](https://github.com/QuivrHQ/MegaParse/commit/b3b80a3a599bbd4bec8ed79bb9ef44c8c7c92789))\n\n## [0.0.32](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.31...megaparse-v0.0.32) (2024-11-01)\n\n\n### Features\n\n* **api:** megaparse under api ([#93](https://github.com/QuivrHQ/MegaParse/issues/93)) ([2edf44b](https://github.com/QuivrHQ/MegaParse/commit/2edf44bd8c09ac7127db74206e463ebe29c68998))\n\n\n### Bug Fixes\n\n* api call error & tests ([#98](https://github.com/QuivrHQ/MegaParse/issues/98)) ([6bf1ce8](https://github.com/QuivrHQ/MegaParse/commit/6bf1ce8c6ed0e4f1e81577973a0fc71f61b10776))\n\n## [0.0.31](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.30...megaparse-v0.0.31) (2024-08-20)\n\n\n### Features\n\n* **pytorch:** cpu only removed ([#88](https://github.com/QuivrHQ/MegaParse/issues/88)) ([6b2fcfa](https://github.com/QuivrHQ/MegaParse/commit/6b2fcfa4413b8a72d398aab57f277dd28ab69c2f))\n\n## [0.0.30](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.29...megaparse-v0.0.30) (2024-08-20)\n\n\n### Features\n\n* **pytorch:** cpu only optional ([#86](https://github.com/QuivrHQ/MegaParse/issues/86)) ([e5d8806](https://github.com/QuivrHQ/MegaParse/commit/e5d8806ee6182de250352ce65ac6cd57c1093494))\n\n## [0.0.29](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.28...megaparse-v0.0.29) (2024-08-18)\n\n\n### Bug Fixes\n\n* **building:** version not working ([#83](https://github.com/QuivrHQ/MegaParse/issues/83)) ([c5e73f6](https://github.com/QuivrHQ/MegaParse/commit/c5e73f6c821424ef277ddd15ddb5b2df48ff7ab2))\n\n## [0.0.28](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.27...megaparse-v0.0.28) (2024-08-16)\n\n\n### Features\n\n* **rye:** added package manager ([#81](https://github.com/QuivrHQ/MegaParse/issues/81)) ([a3a50a3](https://github.com/QuivrHQ/MegaParse/commit/a3a50a3f27d3d9b4d6de4f3415472f8e52710656))\n\n## [0.0.27](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.26...megaparse-v0.0.27) (2024-08-16)\n\n\n### Features\n\n* **unstructured:** increased version ([#78](https://github.com/QuivrHQ/MegaParse/issues/78)) ([eb49cf5](https://github.com/QuivrHQ/MegaParse/commit/eb49cf5e79cd7a38c8212b315a4b64860c35a7b7))\n\n## [0.0.26](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.25...megaparse-v0.0.26) (2024-08-16)\n\n\n### Bug Fixes\n\n* **pycrypto:** being used by an old version of pdfplumber ([#76](https://github.com/QuivrHQ/MegaParse/issues/76)) ([d28f88c](https://github.com/QuivrHQ/MegaParse/commit/d28f88ceb2a722b15c84738f395b3ff4c818a365))\n\n## [0.0.25](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.24...megaparse-v0.0.25) (2024-08-16)\n\n\n### Features\n\n* **rye:** implemented ([#74](https://github.com/QuivrHQ/MegaParse/issues/74)) ([1e9ad8e](https://github.com/QuivrHQ/MegaParse/commit/1e9ad8e0000f28c709d915219fe62c0dbe7fa812))\n\n## [0.0.24](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.23...megaparse-v0.0.24) (2024-07-30)\n\n\n### Features\n\n* async load ([#71](https://github.com/QuivrHQ/MegaParse/issues/71)) ([fbc3e1b](https://github.com/QuivrHQ/MegaParse/commit/fbc3e1b5f504eee9757e15592169ddad9b069f03))\n\n## [0.0.23](https://github.com/QuivrHQ/MegaParse/compare/megaparse-v0.0.22...megaparse-v0.0.23) (2024-07-30)\n\n\n### Features\n\n* megaparse 0.0.22 ([071fd4d](https://github.com/QuivrHQ/MegaParse/commit/071fd4da2e8f0abb58fc66c3cdd87c4ee5cda4d6))\n\n## 0.0.20 (2024-07-10)\n\n## What's Changed\n* add: resolve multiple page problem on llama parse by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/61\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.19...v0.0.20\n\n## 0.0.19 (2024-06-28)\n\n## What's Changed\n* add: choose unstructured strategy by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/57\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.18...v0.0.19\n\n## 0.0.18 (2024-06-28)\n\n## What's Changed\n* fix: add __init__.py by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/54\n* fix: Update README.md by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/56\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.17...v0.0.18\n\n## 0.0.17 (2024-06-27)\n\n## What's Changed\n* markdown by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/48\n* fix:Update README.md by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/49\n* fix: Update README.md by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/50\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.16...v0.0.17\n\n## 0.0.16 (2024-06-27)\n\n## What's Changed\n* Fix: Update README.md by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/47\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.15...v0.0.16\n\n## 0.0.15 (2024-06-26)\n\n## What's Changed\n* add: llm megaparser by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/42\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.14...v0.0.15\n\n## 0.0.14 (2024-06-24)\n\n## What's Changed\n* fix: remove nest asycio by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/40\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.13...v0.0.14\n\n## 0.0.13 (2024-06-24)\n\n## What's Changed\n* fix: use aload_data by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/38\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.12...v0.0.13\n\n## 0.0.12 (2024-06-18)\n\n## What's Changed\n* fix:delete markdownify dependency by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/33\n* fix: fake fix README.md by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/34\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.11...v0.0.12\n\n## 0.0.11 (2024-06-17)\n\n## What's Changed\n* Fix OpenAI key error. Add docstrings. Polish code by @dSupertramp in https://github.com/QuivrHQ/MegaParse/pull/24\n* Fix DOCX reader. Add input tests by @dSupertramp in https://github.com/QuivrHQ/MegaParse/pull/25\n* add: xlsx convertor by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/29\n* add: convert_tab by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/31\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.10...v0.0.11\n\n## 0.0.10 (2024-06-04)\n\n## What's Changed\n* Change from LiteralString to Literal (typing) by @dSupertramp in https://github.com/QuivrHQ/MegaParse/pull/21\n* chore: Add Dockerfile and Makefile for project setup by @StanGirard in https://github.com/QuivrHQ/MegaParse/pull/23\n\n## New Contributors\n* @dSupertramp made their first contribution in https://github.com/QuivrHQ/MegaParse/pull/21\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.9...v0.0.10\n\n## 0.0.9 (2024-06-04)\n\n## What's Changed\n* chore: Update README.md to include optional use of LlamaParse for improved results by @StanGirard in https://github.com/QuivrHQ/MegaParse/pull/19\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.8...v0.0.9\n\n## 0.0.8 (2024-06-04)\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.7...v0.0.8\n\n## 0.0.7 (2024-06-03)\n\n## What's Changed\n* feat: Update benchmark results in README.md by @StanGirard in https://github.com/QuivrHQ/MegaParse/pull/15\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.6...v0.0.7\n\n## 0.0.6 (2024-06-03)\n\n## What's Changed\n* add: gpt cleaner for header and footer by @chloedia in https://github.com/QuivrHQ/MegaParse/pull/13\n\n## New Contributors\n* @chloedia made their first contribution in https://github.com/QuivrHQ/MegaParse/pull/13\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.5...v0.0.6\n\n## 0.0.5 (2024-06-02)\n\n## What's Changed\n* feat: Add instructions for installing poppler and tesseract by @StanGirard in https://github.com/QuivrHQ/MegaParse/pull/10\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.4...v0.0.5\n\n## 0.0.4 (2024-06-02)\n\n## What's Changed\n* add: baseline evaluation by @StanGirard in https://github.com/QuivrHQ/MegaParse/pull/7\n* Add support for Unstructured Parser, improve Table and Image Parsing, and add TOC and Hyperlinks for Docx by @StanGirard in https://github.com/QuivrHQ/MegaParse/pull/9\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.3...v0.0.4\n\n## 0.0.3 (2024-05-30)\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.2...v0.0.3\n\n## 0.0.2 (2024-05-30)\n\n## What's Changed\n* feat: Megaparse example and working by @StanGirard in https://github.com/QuivrHQ/MegaParse/pull/2\n\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.1...v0.0.2\n\n## 0.0.2 (2024-05-30)\n\n**Full Changelog**: https://github.com/QuivrHQ/MegaParse/compare/v0.0.1...v0.0.2\n"
},
{
"path": "Dockerfile",
"content": "FROM python:3.11.10-slim-bullseye\n\nWORKDIR /app\n\n# Install runtime dependencies\nRUN apt-get update && apt-get upgrade && apt-get install -y \\\n libgeos-dev \\\n libcurl4-openssl-dev \\\n libssl-dev \\\n binutils \\\n curl \\\n git \\\n autoconf \\\n automake \\\n build-essential \\\n libtool \\\n python-dev \\\n build-essential \\\n wget \\\n gcc \\\n # Additional dependencies for document handling\n libmagic-dev \\\n poppler-utils \\\n tesseract-ocr \\\n libreoffice \\\n libpq-dev \\\n pandoc && \\\n rm -rf /var/lib/apt/lists/* && apt-get clean\n\nCOPY requirements.lock pyproject.toml README.md ./\nCOPY libs/megaparse/pyproject.toml libs/megaparse/README.md libs/megaparse/\nCOPY libs/megaparse_sdk/pyproject.toml libs/megaparse_sdk/README.md libs/megaparse_sdk/\n\nRUN pip install uv\nRUN uv pip install --no-cache --system -r requirements.lock\n\nRUN playwright install --with-deps\nRUN python3 - -m nltk.downloader all\n\nCOPY . .\n\nRUN uv pip install --no-cache --system /app/libs/megaparse /app/libs/megaparse_sdk\n\nEXPOSE 8000\nCMD [\"uvicorn\", \"megaparse.api.app:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\"]\n"
},
{
"path": "Dockerfile.gpu",
"content": "FROM nvidia/cuda:12.6.3-cudnn-devel-ubuntu20.04\n\nWORKDIR /app\n\nENV UV_COMPILE_BYTECODE=1\nENV UV_NO_CACHE=1\nENV DEBIAN_FRONTEND=noninteractive\n\n# Install runtime dependencies\nRUN apt-get update && apt-get install -y software-properties-common && \\\n add-apt-repository ppa:deadsnakes/ppa && \\\n apt-get update && apt-get install -y \\\n python3.11 \\\n python3.11-dev \\\n libgeos-dev \\\n libcurl4-openssl-dev \\\n libssl-dev \\\n binutils \\\n curl \\\n git \\\n autoconf \\\n automake \\\n libtool \\\n python3-pip \\\n build-essential \\\n wget \\\n gcc \\\n # Additional dependencies for document handling\n libmagic-dev \\\n poppler-utils \\\n tesseract-ocr \\\n libreoffice \\\n libpq-dev \\\n pandoc && \\\n rm -rf /var/lib/apt/lists/* && apt-get clean\n\nRUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 && \\\n update-alternatives --set python3 /usr/bin/python3.11\n\nCOPY requirements.lock pyproject.toml README.md ./\nCOPY libs/megaparse/pyproject.toml libs/megaparse/README.md libs/megaparse/\nCOPY libs/megaparse_sdk/pyproject.toml libs/megaparse_sdk/README.md libs/megaparse_sdk/\n\nRUN curl -LsSf https://astral.sh/uv/install.sh | sh\nENV PATH=\"/root/.local/bin:$PATH\"\nRUN uv pip install --no-cache --system -r requirements.lock\n\nRUN playwright install --with-deps\nRUN python3 - -m nltk.downloader all\n\n# FIXME: causes runtime link issues with onnxruntime_pybind_state.cc:507 unstructured\n# RUN python3 -c \"from unstructured.nlp.tokenize import download_nltk_packages; download_nltk_packages()\" && \\\n# RUN python3 -c \"import nltk; nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger_eng')\"\n\nCOPY . .\n\nRUN uv pip install --no-cache --system /app/libs/megaparse /app/libs/megaparse_sdk\n"
},
{
"path": "LICENSE",
"content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n"
},
{
"path": "Makefile",
"content": ".DEFAULT_TARGET=help\n\n## help: Display list of commands\n.PHONY: help\nhelp:\n\t@echo \"Available commands:\"\n\t@sed -n 's|^##||p' $(MAKEFILE_LIST) | column -t ':' | sed -e 's|^| |'\n\n## dev: Start development environment\n.PHONY: dev\ndev:\n\tDOCKER_BUILDKIT=1 docker compose -f docker-compose.dev.yml up --build\n\n## dev-build: Build development environment without cache\n.PHONY: dev-build\ndev-build:\n\tDOCKER_BUILDKIT=1 docker compose -f docker-compose.dev.yml build --no-cache\n\tDOCKER_BUILDKIT=1 docker compose -f docker-compose.dev.yml up\n\n## prod: Build and start production environment\n.PHONY: prod\nprod:\n\tdocker compose -f docker-compose.yml up --build\n"
},
{
"path": "Pipfile",
"content": "[[source]]\nurl = \"https://pypi.org/simple\"\nverify_ssl = true\nname = \"pypi\"\n\n[packages]\n\n[dev-packages]\n\n[requires]\npython_version = \"3.11\"\n"
},
{
"path": "README.md",
"content": "# MegaParse - Your Parser for every type of documents\n\n
This page shall test if the recorder generates\nscripts with requests in correct order. It includes images in sequential order: stadyn_image1.gif\nthrough stadyn_image10.gif
\n\n\n\n
\nstadyn_image1
\n\n
\nstadyn_image2
\n\n
Open Financial Exchange \nSpecification 1.0
\n\n
February 14, 1997 \n
\n\n
1997 CheckFree Corp., Intuit Inc., Microsoft Corp.\nAll rights reserved \n \n \n
1. Overview 51.1 Introduction 51.1.1 Design Principles 51.2 Open Financial Exchange at\na Glance 71.2.1 Data Transport 71.2.2 Request and Response Model 81.3 Conventions 92.\nStructure 102.1 HTTP Headers 102.2 Open Financial Exchange Headers 112.2.1 The Meaning of\nVersion Numbers 122.3 SGML Details 122.3.1 Compliance 122.3.2 Special Characters 122.4\nOpen Financial Exchange SGML Structure 132.4.1 Overview 132.4.2 Top Level 132.4.3 Messages\n132.4.4 Message Sets and Version Control 142.4.5 Transactions 152.5 The Signon Message Set\n162.5.1 Signon <SONRQ> <SONRS> 162.5.2 PIN Change <PINCHRQ>\n<PINCHRS> 192.5.3 Examples 202.6 External Data Support 202.7 Extensions to Open\nFinancial Exchange 213. Common Aggregates, Elements, and Data Types 223.1 Common\nAggregates 223.1.1 Identifying Financial Institutions and Accounts 223.1.2 Balance Records\n<BAL> 223.1.3 Error Reporting <STATUS> 233.2 Common Elements 243.2.1 Financial\nInstitution Transaction ID <FITID> 243.2.2 Server-Assigned ID <SRVRTID>\n243.2.3 Client-Assigned Transaction UID <TRNUID> 253.2.4 Token <TOKEN> 253.2.5\nTransaction Amount <TRNAMT> 253.2.6 Memo <MEMO> 253.2.7 Date Start and Date\nEnd <DTSTART> <DTEND> 263.3 Common data types 263.3.1 Dates and Times 263.3.2\nAmounts, Prices, and Quantities 283.3.3 Language 283.3.4 Basic data types 284. Security\n294.1 Security Solutions 294.1.1 Determining Security Levels <OFXSEC>\n<TRANSPSEC> 294.2 Channel-Level Security 304.2.1 Security Requirements 304.2.2 Using\nSSL 3.0 in Open Financial Exchange 304.3 Application-Level Security 314.3.1 Requirements\nfor Application-Layer Security 314.3.2 Using Application-level Encryption in Open\nFinancial Exchange 325. International Support 335.1 Language and Encoding 335.2 Currency\n<CURDEF> <CURRENCY> <ORIGCURRENCY> 335.3 Country-Specific Tag Values\n346. Data Synchronization 356.1 Overview 356.2 Background 356.3 Data Synchronization\nApproach 366.4 Data Synchronization Specifics 376.5 Conflict Detection and Resolution\n396.6 Synchronization vs. Refresh 406.7 Typical Server Architecture for Synchronization\n416.8 Typical Client Processing of Synchronization Results 436.9 Simultaneous Connections\n446.10 Synchronization Alternatives 446.10.1 Lite Synchronization 446.10.2 Relating\nSynchronization and Error Recovery 456.11 Examples 467. FI Profile 487.1 Overview 487.1.1\nMessage Sets 487.1.2 Version Control 497.1.3 Batching and Routing 497.2 Profile Request\n507.3 Profile Response 517.3.1 Message Set 527.3.2 Signon Realms 537.3.3 Status Codes\n537.4 Profile Message Set Profile Information 548. Activation & Account Information\n558.1 Overview 558.2 Approaches to User Sign-Up with Open Financial Exchange 558.3 Users\nand Accounts 568.4 Enrollment and Password Acquisition <ENROLLRQ> <ENROLLRS>\n568.4.1 User IDs 578.4.2 Enrollment Request 578.4.3 Enrollment Response 598.4.4 Enrollment\nStatus Codes 598.4.5 Examples 608.5 Account Information 608.5.1 Request <ACCTINFORQ>\n618.5.2 Response <ACCTINFORS> 618.5.3 Account Information Aggregate <ACCTINFO>\n628.5.4 Status Codes 628.5.5 Examples 638.6 Service Activation 638.6.1 Activation Request\nand Response 648.6.2 Service Activation Synchronization 668.6.3 Examples 668.7 Name and\nAddress Changes <CHGUSERINFORQ> <CHGUSERINFORS> 678.7.1 <CHGUSERINFORQ>\n678.7.2 <CHGUSERINFORS> 688.7.3 Status Codes 688.8 Signup Message Set Profile\nInformation 699. Customer to FI Communication 709.1 The E-Mail Message Set 709.2 E-Mail\nMessages 709.2.1 Regular vs. Specialized E-Mail 719.2.2 Basic <MAIL> Aggregate\n719.2.3 E-Mail <MAILRQ> <MAILRS> 719.2.4 E-Mail Synchronization\n<MAILSYNCRQ> <MAILSYNCRS> 729.2.5 Example 739.3 Get HTML Page 749.3.1 MIME Get\nRequest and Response <GETMIMERQ> <GETMIMERS> 749.3.2 Example 759.4 E-Mail\nMessage Set Profile Information 7610. Recurring Transactions 7710.1 Creating a Recurring\nModel 7710.2 Recurring Instructions <RECURRINST> 7710.2.1 Values for <FREQ>\n7810.2.2 Examples 7910.3 Retrieving Transactions Generated by a Recurring Model 8010.4\nModifying and Canceling Individual Transactions 8010.5 Modifying and Canceling Recurring\nModels 8010.5.1 Examples 81\n\n
Open Financial Exchange is a broad-based framework for exchanging\nfinancial data and instructions between customers and their financial institutions. It\nallows institutions to connect directly to their customers without requiring an\nintermediary. \n \n
\n\n
Open Financial Exchange is an open specification that anyone can\nimplement: any financial institution, transaction processor, software developer or other\nparty. It uses widely accepted open standards for data formatting (such as SGML),\nconnectivity (such as TCP/IP and HTTP), and security (such as SSL).
\n\n
Open Financial Exchange defines the request and response messages used\nby each financial service as well as the common framework and infrastructure to support\nthe communication of those messages. This specification does not describe any specific\nproduct implementation.\n\n
The following principles were used in designing Open Financial Exchange:\n
\n\n
l BroadRange of Financial Activities -\nOpen Financial Exchange provides support for a broad range of financial\nactivities. Open Financial Exchange 1.0 specifies the following services:
\n\n
n Bank statement download
\n\n
n Credit card statement download\n
\n\n
n Funds transfers including\nrecurring transfers
\n\n
n Consumer payments, including\nrecurring payments
\n\n
n Business payments, including\nrecurring payments
\n\n
n Brokerage and mutual fund\nstatement download, including transaction history, current holdings and balances
\n\n
l BroadRange of Financial Institutions -\nOpen Financial Exchange supports communication with a broad range of\nfinancial institutions (FIs), including:
\n\n
n Banks
\n\n
n Brokerage houses
\n\n
n Merchants
\n\n
n Processors
\n\n
n Financial advisors
\n\n
n Government agencies
\n\n
l BroadRange of Front-End applications -\nOpen Financial Exchange supports a broad range of front-end applications\ncovering all types of financial activities running on all types of platforms, including\nWeb-based applications.
\n\n
l Extensible - Open Financial Exchange has been\ndesigned to allow the easy addition of new services. Future versions will include support\nfor many new services.
\n\n
l Open - This specification is publicly available.\nYou can build client and server applications using the Open Financial Exchange protocols\nindependent of any specific technology, product, or company.
\n\n
l Multiple Client Support - Open Financial Exchange\nallows a user to use multiple client applications to access the same data at a financial\ninstitution. With the popularity of the World Wide Web, customers are increasingly more\nlikely to use multiple applications-either desktop-based or Web-based-to perform financial\nactivities. For example, a customer can track personal finances at home with a desktop\napplication and occasionally pay bills while at work with a Web-based application. The use\nof data synchronization to support multiple clients is a key innovation in Open Financial\nExchange.
\n\n
l Robust - Open Financial Exchange will be used for\nexecuting important financial transactions and for communicating important financial\ninformation. Assuring users that transactions are executed and information is correct is\ncrucial. Open Financial Exchange provides robust protocols for error recovery.
\n\n
l Secure - Open Financial Exchange provides a\nframework for building secure online financial services. In Open Financial Exchange,\nsecurity encompasses authentication of the parties involved, as well as secrecy and\nintegrity of the information being exchanged.
\n\n
l Batch & Interactive - The design of request and\nresponse messages in Open Financial Exchange is for use in either batch or interactive\nstyle of communication. Open Financial Exchange provides for applying a single\nauthentication context to multiple requests in order to reduce the overhead of user\nauthentication.
\n\n
l InternationalSupport - Open Financial Exchange is designed to supply financial services throughout\nthe world. It supports multiple currencies, country-specific extensions, and different\nforms of encoding such as UNICODE.
\n\n
l Platform Independent -Open Financial Exchange can\nbe implemented on a wide variety of front-end client devices, including those running\nWindows 3.1, Windows 95, Windows NT, Macintosh, or UNIX. It also supports a wide variety\nof Web-based environments, including those using HTML, Java, JavaScript, or ActiveX.\nSimilarly on the back-end, Open Financial Exchange can be implemented on a wide variety of\nserver systems, including those running UNIX, Windows NT, or OS/2.
\n\n
l Transport Independent - Open Financial Exchange is\nindependent of the data communication protocol used to transport the messages between the\nclient and server computers. Open Financial Exchange 1.0 will use HTTP.\n\n
The design of Open Financial Exchange is as a client and server system.\nAn end-user uses a client application to communicate with a server at a financial\ninstitution. The form of communication is requests from the client to the server and\nresponses from the server back to the client.
\n\n
Open Financial Exchange uses the Internet Protocol (IP) suite to provide\nthe communication channel between a client and a server. IP protocols are the foundation\nof the public Internet and a private network can also use them.\n\n
Clients use the HyperText Transport Protocol (HTTP) to communicate to an\nOpen Financial Exchange server. The World Wide Web throughout uses the same HTTP protocol.\nIn principle, a financial institution can use any off-the-shelf web server to implement\nits support for Open Financial Exchange.
\n\n
To communicate by means of Open Financial Exchange over the Internet,\nthe client must establish an Internet connection. This connection can be a dial-up\nPoint-to-Point Protocol (PPP) connection to an Internet Service Provider (ISP) or a\nconnection over a local area network that has a gateway to the Internet.
\n\n
Clients use the HTTP POST command to send a request to the previously\nacquired Uniform Resource Locator (URL) for the desired financial institution. The URL\npresumably identifies a Common Gateway Interface (CGI) or other process on an FI server\nthat can accept Open Financial Exchange requests and produce a response.
\n\n
The POST identifies the data as being of type application/x-ofx. Use\napplication/x-ofx as the return type as well. Fill in other fields per the HTTP 1.0 spec.\nHere is a typical request:
\n\n
POST http://www.fi.com/ofx.cgi HTTP/1.0\nUser-Agent:MyApp 5.0\nContent-Type: application/x-ofx\nContent-Length: 1032\n\nOFXHEADER:100\nDATA:OFXSGML\nVERSION:100\nSECURITY:1\nENCODING:USASCII\n\n<OFX>\n... Open Financial Exchange requests ...\n</OFX>\n
\n\n
A blank line defines the separation between the HTTP headers and the\nstart of the actual Open Financial Exchange data. A blank line also separates the Open\nFinancial Exchange headers and the actual response. (See Chapter 2, for more information.)\n
\n\n
The structure of a response is similar to the request, with the first\nline containing the standard HTTP result, as shown next. The content length is given in\nbytes.
The basis for Open Financial Exchange is the request and response model.\nOne or more requests can be batched in a single file. This file typically includes a\nsignon request and one or more service-specific requests. An FI server will process all of\nthe requests and return a single response file. This batch model lends itself to Internet\ntransport as well as other off-line transports. Both requests and responses are plain text\nfiles, formatted using a grammar based on Standard Generalized Markup Language (SGML).\nOpen Financial Exchange is syntactically similar to HyperText Markup Language (HTML),\nfeaturing tags to identify and delimit the data. The use of a tagged data format allows\nOpen Financial Exchange to evolve over time while continuing to support older clients and\nservers.
\n\n
Here is a simplified example of an Open Financial Exchange request file.\n(This example does not show the Open Financial Exchange headers and the indentation is\nonly for readability.) For complete details, see the more complete examples throughout\nthis specification.
\n\n
<OFX> <!-- Begin request data --> <SIGNONMSGSRQV1> <SONRQ>\n<!-- Begin signon --> <DTCLIENT>19961029101000 <!-- Oct. 29, 1996, 10:10:00\nam --> <USERID>123-45-6789 <!-- User ID (that is, SSN) -->\n<USERPASS>MyPassword <!-- Password (SSL encrypts whole) -->\n<LANGUAGE>ENG <!-- Language used for text --> <FI> <!-- ID of\nreceiving institution --> <ORG>NCH <!-- Name of ID owner -->\n<FID>1001 <!-- Actual ID --> </FI> <APPID>MyApp <APPVER>0500\n</SONRQ> <!-- End of signon --> </SIGNONMSGSRQV1> <BANKMSGSRQV1>\n<STMTTRNRQ> <!-- First request in file --> <TRNUID>1001 <STMTRQ>\n<!-- Begin statement request --> <BANKACCTFROM> <!-- Identify the account\n--> <BANKID>121099999 <!-- Routing transit or other FI ID -->\n<ACCTID>999988 <!-- Account number --> <ACCTTYPE>CHECKING <!--\nAccount type --> </BANKACCTFROM> <!-- End of account ID --> <INCTRAN>\n<!-- Begin include transaction --> <INCLUDE>Y <!-- Include transactions\n--> </INCTRAN> <!-- End of include transaction --> </STMTRQ> <!--\nEnd of statement request --> </STMTTRNRQ> <!-- End of first request -->\n</BANKMSGSRQV1></OFX> <!-- End of request data -->The\nresponse format follows a similar structure. Although a response such as a statement\nresponse contains all of the details of each transaction, each element is identified using\ntags.
\n\n
The key rule of Open Financial Exchange syntax is that each tag is\neither an element or an aggregate. Data follows its element tag. An aggregate tag begins a\ncompound tag sequence, which must end with a matching tag; for example, <AGGREGATE>\n... </AGGREGATE>.
\n\n
The actual file Open Financial Exchange sends is without any extra white\nspace between tags.\n\n
The conventions used in the detailed descriptions include: \n\n
\n
Required tags are in bold. Regular face indicates tags that are\n optional. Required means that a client will always include a tag in a request, and a\n server must always include a tag in a response.
\n
Italic shows a required or optional aggregate from a set of\n possible aggregates.
\n
Required tags occur once unless noted as one or more in the description,\n in which case the specification allows multiple occurrences.
\n
Optional tags occur once if present unless noted as zero or more in the\n description, in which case the specification allows multiple occurrences.
\n
Allowable specific values are listed, where applicable.
\n
A-n or N-n, specify those values that take general\n alphanumeric or pure numeric type values, where n indicates the maximum size.
\n
References to certain common value types, such as a dollar amount, are by\n name. Chapter 3 lists value types that can be referenced by name.
\n
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<REQUIREDTAG>
\n
Required tag (1 or more)
\n
\n
\n
<REQUIREDTAG2>
\n
Required tag that occurs only once
\n
\n
\n
<OPTIONALTAG>
\n
Optional tag; this particular one can occur multiple times\n (0 or more)
\n
\n
\n
<SPECIFIC>
\n
Values are A, B, and C
\n
\n
\n
<ALPHAVALUE>
\n
Takes an alphanumeric value up to 32 characters, A-32\n
This chapter describes the basic structure of an Open Financial Exchange\nrequest and response. Structure includes headers, basic syntax, and the Signon request and\nresponse. This chapter also describes how Open Financial Exchange encodes external data,\nsuch as bit maps.
\n\n
Open Financial Exchange data consists of some headers plus one or more\nOpen Financial Exchange data blocks. Each block consists of a signon message and zero or\nmore additional messages. When sent over the internet using HTTP, standard HTTP and\nmulti-part MIME headers and formats surround the Open Financial Exchange data. A simple\nfile that contained only Open Financial Exchange data would have the following form:\n
Data delivered by way of HTTP places the standard HTTP result code on\nthe first line. HTTP defines a number of status codes. Servers can return any standard\nHTTP result. However, FIs should expect clients to collapse these codes into the following\nthree cases:
\n\n
\n
\n
Code
\n
Meaning
\n
Action
\n
\n
\n
200
\n
OK
\n
The request was processed and a valid Open Financial\n Exchange result is returned.
\n
\n
\n
400s
\n
Bad request
\n
The request was invalid and was not processed. Clients will\n report an internal error to the user.
\n
\n
\n
500s
\n
Server error
\n
The server is unavailable. Clients should advise the user\n to retry shortly.
\n
\n
\n\n
NOTE: Open Financial Exchange returns a code 400 only if it cannot parse the\nfile. Open Financial Exchange handles content errors such as wrong PIN, or invalid\naccount, by returning a valid Open Financial Exchange response along with code 200.
\n\n
Open Financial Exchange requires the following HTTP standard headers:
\n\n
\n
\n
Code
\n
Value
\n
Explanation
\n
\n
\n
Content-type
\n
application/x-ofx
\n
The MIME type for Open Financial Exchange
\n
\n
\n
Content-length
\n
length
\n
Length of the data after removing HTTP headers
\n
\n
\n\n
\n
\n\n
When responding with multi-part MIME, the main type will be\nmulti-part/x-mixed-replace; \none of the parts will use application/x-ofx.\n\n
The intent of Open Financial Exchange is for use with a variety of\ntransports and to provide sufficient version control capabilities for future expansion. To\nsupport this goal, the contents of an Open Financial Exchange file consist of a simple set\nof headers followed by contents defined by that header. \"File format\" means the\nentire content after removal of any transport headers. The HTTP transport described in\nthis document, means without the HTTP and MIME headers.
\n\n
The Open Financial Exchange headers are in a simple tag:value\nsyntax and terminated by a blank line. Open Financial Exchange always sends headers\nunencrypted, even if there is application-level encryption in use for the remaining\ncontents. The first entry will always be OFXHEADER with a version number. This entry will\nhelp identify the contents as an Open Financial Exchange file, and provides the version of\nthe Open Financial Exchange headers that follow (not of the content itself). For example:\n
\n\n
OFXHEADER:100\n
\n\n
This document defines version 1.0 of the headers to contain at least the\nfollowing additional tags:
The data tag identifies the contents as being in OFX SGML form. VERSION\nidentifies the version type as OFXSGML data. In the case of OFXSGML, it translates to the\nversion of the Document Type Definition (DTD) that it uses for parsing. The ENCODING and\nCHARSET tags define the interpretation of the character data. See Chapter 5,\n\"International Support\" for more information on these tags. Chapter 4 describes\nthe security tag. A future version of this specification will define compression.
\n\n
Open Financial Exchange uses OLDFILEUID and NEWFILEUID to support error\nrecovery. They are not present when clients are not requesting error recovery. (See\nChapter 6, \"Data Synchronization\")
\n\n
A blank line follows the last tag. Then (for type OFXSGML), the\nSGML-readable data begins with the <OFX> tag.
\n\n
NOTE: Here, VERSION provides the overall version of the DTD.\nThe <OFX> block describes the specific message set versions used, shown later in\nthis chapter.\n\n
The OFXHEADER value should only change its major number if an existing\nclient is unable to process the new header. This can occur because of a complete syntax\nchange in a header, or a significant change in the semantics of an existing tag-not the\nentire response. You can add new tags as long as clients can function without\nunderstanding them.
\n\n
You should add new values for a data tag only when you introduce an\nentirely new syntax. In the case of OFXSGML, a new syntax would have to be non-SGML\ncompliant to warrant a new data value. It is possible that there will be more than one\nsyntax in use at the same time to meet different needs.
\n\n
The intent of the header version tag is to identify syntactic changes.\nIn the case of OFXSGML, this corresponds to the DTD. Purely for identification purposes,\neach change will increment the minor number of the version tag. If you introduce an\nincompatible change so that an older DTD can not parse the file, the major number will\nchange. See the general discussion of message sets and version control, later in this\nchapter.\n\n
SGML is the basis for Open Financial Exchange. There is a DTD that\nformally defines the SGML wire format. However, Open Financial Exchange is not completely\nSGML-compliant because the specification allows unrecognized tags to be present. It\nrequires clients and servers to skip over the unrecognized material. That is, if\n<XYZ>qqq</XYZ> appeared and a client or server cannot recognize <XYZ>,\nthe server should ignore that tag and its enclosed data. A fully-compliant SGML parser\nwould not validate an Open Financial Exchange document if it contained any tags\nthat the DTD does not define.
\n\n
Although SGML is the basis for the specification, and the specification\nis largely compliant with SGML, do not assume Open Financial Exchange supports any SGML\nfeatures not documented in this specification. The intent is to allow parsing to be as\nsimple as possible, while retaining compatibility with the SGML world.\n\n
The following characters are special to SGML. Use the given alternative\nsequence to represent them:
\n\n
\n
\n
Character
\n
Escape sequence
\n
\n
\n
< (less than)
\n
<
\n
\n
\n
> (greater than)
\n
>
\n
\n
\n
& (ampersand)
\n
&
\n
\n
\n\n
For example, the string \"AT&T\" encodes \"AT&T.\"
\n\n
A special case applies in specific tags that can accept HTML-formatted strings, such as\ne-mail records. These accept SGML marked section syntax to hide the HTML from the Open\nFinancial Exchange parser. You must prefix strings with \"<![ CDATA [\"and\nsuffixed with\"]]>.\" Within these bounds, treat the above characters literally\nwithout an escape. See the Chapter 9 for an example.\n\n
Open Financial Exchange hierarchically organizes request and response\nblocks:
\n\n
Top Level <OFX> \nMessage Set and Version <XXXMSGSVn> \nSynchronization Wrappers <YYYSYNCRQ>, <YYYSYNCRS> \nTransaction Wrappers <YYYTRNRQ>, <YYYTRNRS> \nSpecific requests and responses
\n\n
The following sections describe each of these levels.\n\n
An Open Financial Exchange request or response has the following\ntop-level form:
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<OFX>
\n
Opening tag
\n
\n
\n
... Open Financial Exchange requests or responses ...\n
\n
0 or more transaction requests and responses inside\n appropriate message set aggregates
\n
\n
\n
</OFX>
\n
Closing tag for the Open Financial Exchange record
\n
\n
\n\n
This chapter specifies the order of requests and responses.
\n\n
A single file can contain multiple <OFX> ... </OFX> blocks. A typical use\nof multiple blocks is to request in a single file information associated with different\nusers.\n\n
A message is the unit of work in Open Financial Exchange. It refers to a\nrequest and response pair, and the status codes associated with that response. For\nexample, the message to download a bank statement consists of the request <STMTRQ>\nand the response <STMTRS>. In addition, with the exception of the signon message,\neach message includes a transaction wrapper. These aggregates add a transaction\nunique ID <TRNUID>, and for responses, a <STATUS> aggregate, to the basic\nrequest and response.
\n\n
For messages subject to synchronization (see Chapter 6), a third layer\nof aggregates is also part of a message definition: a synchronization request and\nresponse. These add a token and, in some cases, other information to the transactions.
\n\n
Open Financial Exchange uses the following naming where the XXX\nmessage includes:\n\n
\n
Basic request <XXXRQ> and response <XXXRS>\n
\n
Transaction wrapper <XXXTRNRQ> and <XXXTRNRS>\n
\n
If needed, synchronization wrapper <XXXSYNCRQ> and\n <XXXSYNCRS>
\n
\n\n
In a few cases, a small number of related basic requests and responses\nshare a transaction and synchronization wrapper. The term message will still apply to each\nrequest and response; only the naming scheme will not hold in those cases.\n\n
Message sets are collections of messages. Generally they form all or\npart of what a user would consider a service, something for which they might have\nsigned up, such as \"banking.\" Message sets are the basis of version control,\nrouting, and security. They are also the basis for the required ordering in Open Financial\nExchange files.
\n\n
Within an Open Financial Exchange block, Open Financial Exchange\norganizes messages by message set. A message set can appear at most once within an Open\nFinancial Exchange block. All messages from a message set must be from the same version of\nthat message set.
\n\n
For each message set of XXX and version n, there exists an\naggregate named <XXXMSGSVn>. (Compare with <XXXMSGSETVn>\nin Chapter 7.) All of the messages from that message set must be inside the appropriate\nmessage set aggregate. In the following example, the Open Financial Exchange block\ncontains a signon request inside the signon message set, and two statement requests and a\ntransfer request inside the bank message set.
\n\n
<OFX>\n\t<SIGNONMSGSRQV1>\t<!-- Signon message set -->\n\t\t<SONRQ>\t\t\t\t<!-- Signon message -->\n\t\t...\n\t\t</SONRQ>\n\t</SIGNONMSGSRQV1>\n\n\t<BANKMSGSRQV1>\t\t<!-- Banking message set -->\n\t\t<STMTTRNRQ>\t\t<!-- Statement request -->\n\t\t...\n\t\t</STMTTRNRQ>\n\t\t<STMTTRNRQ>\t\t<!-- Another stmt request -->\n\t\t...\n\t\t</STMTTRNRQ>\n\t\t<INTRATRNRQ>\t\t<!-- Intra-bank transfer request -->\n\t\t...\n\t\t</INTRATRNRQ>\n\t</BANKMSGSRQV1>\n</OFX>\n
\n\n
Message sets, if used at all, must appear in the following order:\n\n
\n
Signon
\n
Signup
\n
Banking
\n
Credit card statements
\n
Investment statements
\n
Interbank funds transfers
\n
Wire funds transfers
\n
Payments
\n
General e-mail
\n
Investment security list
\n
FI Profile
\n
\n\n
The definition of each message set can further prescribe an order of its\nmessages within that message set.\n\n
Other than the signon message, each request is made as a transaction.\nTransactions contain a client-assigned globally unique ID, optional client-supplied\npass-back data, and then the record for the specific request. A transaction similarly\nwraps each response. The response transaction returns the client ID sent in the request,\nalong with a status message, the pass-back data if present, and the specific response\nrecord. This technique allows a client to track responses against requests.
\n\n
The <STATUS> aggregate, defined in Chapter 3, provides feedback on\nthe processing of the request. If the <SEVERITY> of the status is ERROR, the server\nprovides no specific response record. Otherwise, the response will be complete even though\nsome warning might have occurred.
\n\n
Clients can send additional information in <CLTCOOKIE> that\nservers will return in the response. This allows clients that do not maintain state, and\nthus do not save TRNUIDs, to cause some additional descriptive information to be present\nin the response. For example, a client might identify a request as relating to a user or a\nspouse.
\n\n
In some countries some transactions require a customer-supplied\nauthorization number for each transaction. In those countries, the <TAN> element\nprovides the means to pass this information to servers. As Open Financial Exchange is\nimplemented in each country, the specification will define the specific requirements for\nthe use of <TAN> in each country.
\n\n
A typical request is as follows:
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<XXXTRNRQ>
\n
Transaction-request aggregate
\n
\n
\n
<TRNUID>
\n
Client-assigned globally unique ID for this transaction trnuid\n
\n
\n
\n
<CLTCOOKIE>
\n
Data to be echoed in the transaction response A-32\n
\n
\n
\n
<TAN>
\n
Transaction authorization number; used in some countries\n with some types of transactions. Country-specific documentation will define messages that\n require a TAN, A-80
\n
\n
\n
specific request
\n
Aggregate for the specific request
\n
\n
\n
</XXXTRNRQ>
\n
\n
\n
\n\n
A typical response is as follows: \n
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<XXXTRNRS>
\n
Transaction-response aggregate
\n
\n
\n
<TRNUID>
\n
Client-assigned globally unique ID for this transaction, trnuid\n
\n
\n
\n
<CLTCOOKIE>
\n
Client-provided data, REQUIRED if provided in\n request, A-32
The Signon message set includes the signon message and the PIN change\nmessage, and must appear in that order. The <SIGNONMSGSRQV1> and\n<SIGNONMSGSRSV1> aggregates wrap the message.\n\n
The signon record identifies and authenticates a user to an FI. It also\nincludes information about the application making the request, because some services might\nbe appropriate only for certain clients. Every Open Financial Exchange request contains\nexactly one <SONRQ>. Every response must contain exactly one <SONRS> record.
\n\n
Use of Open Financial Exchange presumes that FIs authenticate each\ncustomer and then give the customer access to one or more accounts or services. If\npasswords are specific to individual services or accounts, a separate Open Financial\nExchange request will be made for each distinct user ID or password required. This will\nnot necessarily be in a manner visible to the user. Note that some situations, such as\njoint accounts or business accounts, will have multiple user IDs and multiple passwords\nthat can access the same account.
\n\n
FIs assign user IDs for the customer. It can be the customer's social\nsecurity number, but the client will not make any assumptions about the syntax of the ID,\nadd check-digits, or do similar processing.
\n\n
To improve server efficiency in handling a series of Open Financial\nExchange request files sent over a short period of time, clients can request that a server\nreturn a <USERKEY> in the signon response. If the server provide a user key, clients\nwill send the <USERKEY> in instead of the user ID and password in subsequent\nsessions, until the <USERKEY> expires. This allows servers to authenticate\nsubsequent requests more quickly.
\n\n
The client returns <SESSCOOKIE> if it sent one in a previous\n<SONRS>. Servers can use this value to track client usage but cannot assume that all\nrequests come from a single client, nor can they deny service if they did not expect the\nreturned cookie. Use of a backup file, for example, would lead to an unexpected\n<SESSCOOKIE> value that should nevertheless not stop a user from connecting.
\n\n
Servers can request that a consumer change his or her password by\nreturning status code 15000. Servers should keep in mind that only one status code can be\nreturned. If the current signon response status should be 15500 (invalid ID or password),\nthe request to change password will need to wait until an otherwise successful signon is\nachieved.\n\n
\n
Record Request <SONRQ>
\n\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<SONRQ>
\n
Record- request aggregate
\n
\n
\n
<DTCLIENT>
\n
Date and time of the request from the client computer, datetime\n
\n
\n
\n
<USERID>
\n
User identification string. Use <USERID> &\n <USERPASS>, or <USERKEY>, but not both; A-32
\n
\n
\n
<USERPASS>
\n
User password on server - either <USERID> &\n <USERPASS> are used, or <USERKEY>, but not both; A-32
\n
\n
\n
<USERKEY>
\n
Login using previously authenticated context - use\n <USERID> & <USERPASS>, or <USERKEY>, but not both; A-64\n
\n
\n
\n
<GENUSERKEY>
\n
Request server to return a USERKEY for future use, Boolean\n
\n
\n
\n
<LANGUAGE>
\n
Requested language for text responses, language\n
\n
\n
\n
<SESSCOOKIE>
\n
Session cookie, value received in previous <SONRS>,\n not sent if first login or if none sent by FI A-1000
\n
\n
\n
<FI>
\n
Financial-Institution-identification aggregate
\n
\n
\n
</FI>
\n
\n
\n
\n
<APPID>
\n
ID of client application, A-5
\n
\n
\n
<APPVER>
\n
Version of client application, N-4 (6.00 encoded as\n 0600)
\n
\n
\n
</SONRQ>
\n
\n
\n
\n\n\n
Response <SONRS>
\n\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<SONRS>
\n
Record-response aggregate
\n
\n
\n
<STATUS>
\n
Status aggregate, see list of possible code values
\n
\n
\n
<DTSERVER>
\n
Date and time of the server response, datetime\n
\n
\n
\n
<USERKEY>
\n
Use user key that instead of USERID and USERPASS for\n subsequent requests. TSKEYEXPIRE can limit lifetime
\n
\n
\n
<TSKEYEXPIRE>
\n
Date and time that USERKEY expires
\n
\n
\n
<LANGUAGE>
\n
Language used in text responses, language
\n
\n
\n
<DTPROFUP>
\n
Date and time of last update to profile information for any\n service supported by this FI (see Chapter 7), datetime
\n
\n
\n
<DTACCTUP>
\n
Date and time of last update to account information (see\n Chapter 8), datetime
\n
\n
\n
<FI>
\n
Financial-Institution-identification aggregate
\n
\n
\n
</FI>
\n
\n
\n
\n
<SESSCOOKIE>
\n
Session cookie that the client should return on the next\n <SONRQ> \n A-1000
\n
\n
\n
</SONRS>
\n
\n
\n
\n\n
List of status code values for the <CODE> element of <STATUS>: \n
\n\n
\n
\n
Value
\n
Meaning
\n
\n
\n
0
\n
Success (INFO)
\n
\n
\n
2000
\n
General error (ERROR)
\n
\n
\n
15000
\n
Must change PIN (INFO)
\n
\n
\n
15500
\n
Signon (for example, user ID or password) invalid (ERROR)\n
\n
\n
\n
15501
\n
Customer account already in use (ERROR)
\n
\n
\n
15502
\n
PIN Lockout (ERROR)
\n
\n
\n\n\n
Financial Institution ID <FI>
\n\n\n
Some service providers support multiple FIs, and assign each FI an ID.\nThe signon allows clients to pass this information along, so that providers will know to\nwhich FI the user is actually doing a signon.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<FI>
\n
FI-record aggregate
\n
\n
\n
<ORG>
\n
Organization defining this FI name space, A-32\n
\n
\n
\n
<FID>
\n
Financial Institution ID (unique within <ORG>), A-32\n
The signon sends a request to change a customer password as a separate\nrequest. The transaction request <PINCHTRNRQ> aggregate contains <PINCHRQ>.\nResponses are also inside transaction responses <PINCHTRNRS>.
\n\n
Password changes pose a special problem for error recovery. If the\nclient does not receive a response, it does not know whether the password change was\nsuccessful or not. Open Financial Exchange recommends that servers accept either the old\npassword or the new password on the connection following the one containing a password\nchange. The password used becomes the new password.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<PINCHRQ>
\n
PIN-change-request aggregate
\n
\n
\n
<USERID>
\n
User identification string. Often a social security number,\n but if so, does not include any check digits, A-32
\n
\n
\n
<NEWUSERPASS>
\n
New user password, A-32
\n
\n
\n
</PINCHRQ>
\n
\n
\n
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<PINCHRS>
\n
PIN-change-response aggregate
\n
\n
\n
<USERID>
\n
User identification string. Often a social security number,\n but if so, does not include any check digits, A-32
\n
\n
\n
<DTCHANGED>
\n
Date and time the password was changed, datetime\n
\n
\n
\n
</PINCHRS>
\n
\n
\n
\n\n\n
Status Code Values for the <CODE> Element of\n <STATUS>
Some data, such as binary data, cannot be easily sent directly within\nSGML. For these situations, the specification will define a tag that contains a reference\nto some external data. The way that clients pick up the external data depends on the\ntransport used. For the HTTP-based transport described in this document, servers can send\nthe data in one of two ways:\n\n
\n
Send the same response, using multi-part MIME types to separate the\n response into basic \n Open Financial Exchange and one or more external data files
\n
Client can make a separate HTTP get against the supplied URL, if it\n really needs the data
\n
\n\n
For example, to retrieve a logo, a <GETMIMERS> might answer a\n<GETMIMERQ> as follows:
An organization that provides a customized client and server that\ncommunicate by means of \nOpen Financial Exchange might wish to add new requests and responses or even specific\nelements to existing requests and responses. To ensure that each organization can extend\nthe specification without the risk of conflict, Open Financial Exchange defines a style of\ntag naming that lets each organization have its own name space.
\n\n
Organizations can register a specific tag name prefix. (The specific\nprocedure or organization to manage this registration will be detailed at a later time.)\nIf an organization registers \"ABC,\" then they can safely add new tags named\n<ABC.SOMETHING> without\n\n
\n
Colliding with another party wishing to extend the specification
\n
Confusing a client or server that does not support the extension
\n
\n\n
The extensions are not considered proprietary. An organization is free\nto publish their extensions and encourage client and server implementers to support them.\n
\n\n
All tag names that do not contain a period (.) are reserved for use in\nfuture versions of the core \nOpen Financial Exchange specification. \n\n\n
Open Financial Exchange does not provide a universal space for\nidentifying financial institutions, accounts, or types of accounts. The way to identify an\nFI and an account at that FI depends on the service. For information about\nservice-specific ID aggregates, see Chapters 11, 12, and 13 on banking, payments, and\ninvestments.\n\n
Several responses allow FIs to send an arbitrary set of balance\ninformation as part of a response, for example a bank statement download. FIs might want\nto send information on outstanding balances, payment dates, interest rates, and so forth.\nBalances can report the date the given balance reflects in <DTASOF>.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<BAL>
\n
Balance-response aggregate
\n
\n
\n
<NAME>
\n
Balance name, A-20
\n
\n
\n
<DESC>
\n
Balance description, A-80
\n
\n
\n
<BALTYPE>
\n
Balance type. \n DOLLAR = dollar (value formatted DDDDcc) \n PERCENT = percentage (value formatted XXXX.YYYY) \n NUMBER = number (value formatted as is)
\n
\n
\n
<VALUE>
\n
Balance value. \n Interpretation depends on <BALTYPE> field, N-20
\n
\n
\n
<CURRENCY>
\n
If dollar formatting, can optionally include currency\n
To provide as much feedback as possible to clients and their users, Open\nFinancial Exchange defines a <STATUS> aggregate. The most important element is the\ncode that identifies the error. Each response defines the codes it uses. Codes 0 through\n2999 have common meanings in all Open Financial Exchange transactions. Codes from 3000 and\nup have meanings specific to each transaction.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<STATUS>
\n
Error-reporting aggregate.
\n
\n
\n
<CODE>
\n
Error code, N-6
\n
\n
\n
<SEVERITY>
\n
Severity of the error: \n INFO = Informational only \n WARN = Some problem with the request occurred but valid response still present \n ERROR = A problem severe enough that response could not be made
\n
\n
\n
<MESSAGE>
\n
A textual explanation from the FI. Note that clients will\n generally have messages of their own for each error ID. Use this tag only to provide more\n details or for the General errors.
\n
\n
\n
</STATUS>
\n
\n
\n
\n\n
\n\n
\nstadyn_image6
\n\n
\nstadyn_image7
\n\n
For general errors, the server can respond with one of the following <CODE>\nvalues. However, not all codes are possible in a specific context.
\n\n
\n
\n
Code
\n
Meaning
\n
\n
\n
0
\n
Success (INFO)
\n
\n
\n
2000
\n
General error (ERROR)
\n
\n
\n
2021
\n
Unsupported version (ERROR)
\n
\n
\n\n
NOTE: Clients will generally have error messages based on <CODE>.\nTherefore, do not use <MESSAGE> to replace that text. Use <MESSAGE> only to\nexplain an error not well described by one of the defined CODEs, or to provide some\nadditional information.\n\n
This section defines elements used in several services of Open Financial\nExchange. The format of the value is either alphanumeric (A-n)or numeric\n(N-n) with a maximum length n; or as a named type. Section 3.3 describes the\nnamed types.\n\n
An FI assigns an <FITID> to uniquely identify a transaction. Its\nprimary purpose is to allow a client to detect duplicate responses. Open Financial\nExchange intends <FITID> for use in statement download applications, where every\ntransaction requires a unique ID; not just those that are client-originated or\nserver-originated.
\n\n
FITIDs must be unique within the scope of the requested transactions\n(that is, within an account) but need not be sequential or even increasing. Clients should\nbe aware that FITIDs are not unique across FIs. If a client performs the same type of\nrequest within the same scope at two different FIs, clients will need to use FI + account\n+ <FITID> as a unique key in a client database.
\n\n
Usage: Bank statement download, investment statement download\n\n
A <SRVRTID> is a server-assigned ID. It should remain constant\nthroughout the lifetime of the object on the server. The client will consider the SRVRTID\nas its \"receipt\" or confirmation and will use this ID in any subsequent requests\nto change, delete, or inquire about this object.
\n\n
Where the context allows, it is possible for a server to use the same value\nfor a given server object for both <SRVRTID> and <FITID>, but the client\nwill not know this. SRVRTIDs need be unique only within the scope of the requests and\nresponses they apply to, such as an account number. Like <FITID>, a <SRVRTID>\nis not unique across FIs and clients might need to use FI + <SRVRTID> if a client\nrequires a unique key.
Open Financial Exchange uses TRNUIDs to identify transactions,\nspecifically <XXXTRNRQ>. Clients expect a server to return the same\n<TRNUID> in the corresponding response and can use this to match up requests and\nresponses. Servers can use TRNUIDs to reject duplicate requests. Because multiple clients\nmight be generating requests to the same server, transaction IDs need to be unique across\nclients. Thus, <TRNUID> must be a globally unique ID.
\n\n
The Open Software Foundation Distributed Computing Environment standards\nspecify a 36-character hexadecimal encoding of a 128-bit number and an algorithm to\ngenerate it. Clients are free to use their own algorithm, to use smaller TRNUIDs, or to\nrelax the uniqueness requirements if in their particular application it makes sense.\nHowever, it is RECOMMENDED that clients allow for the full 36 characters in\nresponses to work better with other clients.
Open Financial Exchange uses <TOKEN> as part of data\nsynchronization requests to identify the point in history that the client has already\nreceived data, and in responses to identify the server's current end of history. See\nChapter 6, \"Data Synchronization,\" for more information.
\n\n
<TOKEN> is unique within an FI and the scope of the\nsynchronization request. For example, if the synchronization request includes an account\nID, the <TOKEN> needs be unique only within an account. Servers are free to use a\n<TOKEN> that is unique across the entire FI. Clients must save separate\n<TOKEN>s for each account, FI, and type of synchronization request.
\n\n
Usage: All synchronization requests and responses\n\n
Open Financial Exchange uses these tags in requests to provide guidance\nto the FI about the range of response that is desired. It also uses them in responses to\nlet clients know what the FI was actually able to produce.
\n\n
In requests, the following rules apply:\n\n
\n
If <DTSTART> is absent, the client is requesting all available\n history (up to the <DTEND>, if specified). Otherwise, it indicates the inclusive\n date and time in history where the client expects servers to start sending information.\n
\n
If <DTEND> is absent, the client is requesting all available\n history (starting from <DTSTART>, if specified). Otherwise, it indicates the exclusive\n date and time in history where the client expects servers to stop sending information.\n
\n
\n\n
In responses, the following rules apply:\n\n
\n
<DTSTART> is the date and time where the server began looking\n for information, not necessarily the date of the earliest returned information. If the\n response <DTSTART> is later than the requested <DTSTART>, clients can infer\n that the user has not signed on frequently enough to ensure that the client has retrieved\n all information. If the user has been calling frequently enough, <DTSTART> in the\n response will match <DTSTART> in the request.
\n
<DTEND> is the date and time that, if used by the client as the\n next requested <DTSTART>, it would pick up exactly where the current response left\n off. It is the exclusive date and time in history where the server stopped looking\n for information, based on the request <DTEND> rules.
\n
\n\n
In all cases, servers are REQUIRED to use a \"system add\ndatetime\" as the basis for deciding which details match the requested date range. For\nexample, if an FI posts a transaction dated Jan 3 to a user's account on Jan 5, and a\nclient connects on Jan 4 and again on Jan 6, the server is REQUIRED to return that\nJan 3 dated transaction to the client when it calls on Jan 6.
\n\n
Usage: Bank statement download, investment statement download\n\n
There is one format for representing dates, times, and time zones. The\ncomplete form is:
\n\n
YYYYMMDDHHMMSS.XXX[gmt offset:tz name]
\n\n
For example, \"19961005132200.1234[-5:EST]\" represents October\n5, 1996, at 1:22 and 124 milliseconds p.m., in Eastern Standard Time. This is the same as\n6:22 p.m. Greenwich Mean Time (GMT).
\n\n
Tags specified as type date and generally starting with the\nletters \"DT\" will accept a fully formatted date-time-timezone as specified\nabove. They will also accept values with fields omitted from the right. They assume the\nfollowing defaults if a field is missing:\n\n
\n
YYYYMMDD: 12:00 midnight (the start of the day), GMT
\n
YYYYMMDDHHMMSS: GMT
\n
YYYYMMDDHHMMSS.XXX: GMT
\n
YYYYMMDDHHMMSS.XXX[-0500:EST]: Fully qualified
\n
\n\n
Open Financial Exchange identifies elements that require a time as\nhaving type timestamp and their tag name will start with the prefix TS. The\ntimezone and milliseconds are still optional, and will default to GMT.
\n\n
Take care when specifying an ending date without a time. If the last\ntransaction returned for a bank statement download was Jan 5 1996 10:46 a.m. and if the\n<DTEND> was given as just Jan 5, the transactions on Jan 5 would be resent. If\nresults are only available daily, then just using dates and not times will work correctly.\n
\n\n
NOTE: Open Financial Exchange does not require servers or\nclients to use the full precision specified. However, they are REQUIRED to\naccept any of these forms without complaint.
\n\n
Some services extend the general notion of a date by adding\nspecial values, such as \"TODAY.\" These special values are called \"smart\ndates.\" Specific requests indicate when to use these extra values, and list the tag\nas having a special data type.\n\n
\n
Time Zone Issues
\n\n\n
Several issues arise when a customer and the FI are not in the same time\nzone, or when a customer moves a computer into new time zones. In addition, it is\ngenerally unsafe to assume that computer users have correctly set their time or timezone.\n
\n\n
Although most transactions are not sensitive to the exact time, they\noften are sensitive to the date. In some cases, time zone errors lead to actions occurring\non a different date than intended by the customer. For this reason, servers should always\nuse a complete local time plus GMT offset in any datetime values in a response. If a\ncustomer's request is for 5 p.m. EST, and a server in Europe responds with 1 a.m. MET the\nnext day, a smart client can choose to warn the customer about the date shift.
\n\n
Clients that maintain local state, especially of long-lived server\nobjects, should be careful how they store datetime values. If a customer initiates a\nrepeating transaction for 5 p.m. EST, then moves to a new time zone, the customer might\nhave intended that the transaction remain 5 p.m. in the new local time, requiring a change\nrequest to be sent to the server. If, however, they intended it to remain fixed in server\ntime, this would require a change in the local time stored in the client.\n\n
Unless otherwise noted in the specification, Open Financial Exchange\nalways signs amounts and quantities from the perspective of the customer. Some typically\nnegative amounts:\n\n
\n
Investment buy amount, investment sell quantity
\n
Bank statement debit amounts, checks, fees
\n
Credit card purchases
\n
Margin balance (unless the FI owes the client money)
\n
\n\n
Some typically positive amounts:\n\n
\n
Investment sell amount, investment buy quantity
\n
Bank statement credits
\n
Credit card payments
\n
Ledger balance (unless the account is overdrawn)
\n
\n\n
Amount: All amount-valued tags are sent with a decimal point or\ncomma, as in \"XXXX.XX.\" There should not be any punctuation separating\nthousands, millions, and so forth. The maximum value accepted depends on the client.\n
\n\n
Quantity: Use decimal notation.
\n\n
Unitprice: Use decimal notation. Unless specifically noted,\nprices should always be positive.
\n\n
Rate: Use decimal notation, with the rate specified out of 100%.\nFor example, 5.2 is 5.2%.
\n\n
Some services define special values, such as INFLATION, which you can\nuse instead of a designated value. Open Financial Exchange refers to these as \"smart\ntypes,\" and identifies them in the specification.\n\n
Open Financial Exchange identifies human-readable language-for such\nthings as status messages and e-mail-with a three-letter code based on ISO-639.\n\n
Open Financial Exchange carries financial information over the Internet\nin such a way to provide privacy, message integrity, and authentication for applications\nat the appropriate level. Each service within Open Financial Exchange requires a certain\nlevel of security. Online banking and payments require strong secrecy, whereas stock\nquotations consist of publicly available information and consequently have a much weaker\nsecrecy requirement.
\n\n
Some Internet protocols, such as HTTPS (which uses Secure Socket Layer\nversion 3, SSLv3), provide channel-level security. When the security requirement exceeds\nthat provided by the channel, you must use an application-level protocol.
Secure Socket Layer version 3 (SSLv3) provides channel-level security in\nOpen Financial Exchange. SSLv3 provides confidentiality, message integrity, and implicit\nauthentication. In Open Financial Exchange 1.0, channel-level security using SSLv3 is the\nprimary form of security.\n\n
Open Financial Exchange provides a method to exchange financial\ninformation over public networks. This necessitates strong security facilities and careful\nprotocol design. The most commonly used facility, and trusted method for accomplishing\nmany of these goals is SSL. The following sub-sections describe the most prominent\nrequirements for security and how Secure Socket Layer (SSL) addresses these.\n\n
\n
Privacy, Authentication, and Message Integrity
\n\n\n
SSL provides a range of strong encryption methods for insuring\nconfidentiality, and strong measures to insure that messages are not altered as they\npropagate over the Internet. User authentication is usually addressed at the application\nlayer, not within SSL. Servers are configured with public key certificates that client\napplication software verifies. This provides some measure of server authentication.\nTesting certificate revocation lists is not commonly performed. However, as these\nfacilities emerge, client software will be written to support this need. \n\n
\n
Facilities for Authorization
\n\n\n
Open Financial Exchange messages typically provide user ID and password\nso that a service provider can authenticate the user. Once a system authenticates a user,\nthe service provider must insure that the user is authorized to perform the requested\nactions. For example, the service provider must decide if the specified user is authorized\nto perform a transfer from the specified account. The service provider must also determine\nwhether the user has exceeded allowed limits on withdrawals, whether the activity on this\naccount is unusual given past history, and other context-sensitive issues.\n\n
SSL version 3.0 provides a set of widely and commonly accepted methods\nfor securing Internet transactions. These common methods within SSL are called\nCipherSuites. You can secure applications appropriately within SSL by specifying an\nordered sequence of preferred CipherSuites (highest preference listed first). Servers\nselect the strongest supported CipherSuite from the list provided by the client.
\n\n
NOTE: Passing username and password pairs in a weakly\nencrypted channel exposes this information to cryptographic attack. When implementing Open\nFinancial Exchange, use the strongest available ciphers.
\n\n
You should not use the following CipherSuites because they are\nvulnerable to man-in-the-middle attacks during Open Financial Exchange message exchanges:\n\n
While strong channel-level security is sufficient for the current suite\nof Open Financial Exchange transactions, there are features that channel security does not\nprovide. These include (but are not limited to) data signing, non-repudiation, rational\ncertificate management and revocation, and trust proxy. Where the trust model for an\napplication requires such features to conduct the transaction safely, Open Financial\nExchange stipulates the use of an application-level protocol. A future implementation\nguide will publish this protocol.
\n\n
The standard method for providing application-level security is to rely\nupon the RSA Public Key Cryptography Standard (PKCS) message format. The PKCS #7 standard\nspecifies a message format that is both cryptographically strong and flexible enough to\nprovide sufficient facilities for evolution. \n\n
RSA Public Key Cryptography Standard #7 (PKCS#7) defines a rich set of\nmessage formats for securely exchanging information over public networks. These message\nformats provide for encrypting data using a combination of cryptographic techniques to\nleverage manageability of public key cryptography. It also utilizes the speed of block\nciphers into a hybrid, which exploits the best properties of each.
\n\n
PKCS#7 message encryption provides privacy. A digitally signed message\n(or applying HMAC) insures message integrity.
\n\n
Use one of the following to define PKCS#7 messages: Data, Digitally\nSigned-Data, Enveloped-Data, or Digitally Signed and Enveloped-Data (also referred to as\nSealed-Data). Open Financial Exchange can use Digested-Data, which digests application\ndata before it embeds data within an Enveloped-Data object. However, it should not\ntransmit this data over public networks without encryption applied.\n\n
\n
Facilities for Authorization
\n\n\n
As stated previously in the section 4.2, authentication and\nauthorization are the responsibility of the service provider. Open Financial Exchange\nmessages contain the information to enable authentication and authorization decisions.\nWith application-level security that uses a digitally signed format, the verification of\nthat signature provides an additional method of authenticating the user.\n\n
Open Financial Exchange applications requiring a sophisticated trust\nmodel require more facilities than those provided by SSL. If an Open Financial Exchange\napplication requires only point-to-point security, SSL version 3.0 provides adequate\nfacilities for message security. However, if the application requires more directed,\nspecific forms of security, then use the appropriate PKCS#7 message formats for the\napplication. An example of this might be a stock trading application issuing orders whose\nvalues demand that the security level be high, and where Open Financial Exchange treats\nthe message with special handling instructions.
\n\n
Recommended cryptographic techniques for Open Financial Exchange\napplication security are: \n\n
\n
RC4 for bulk encryption (using 40 bits for exportable applications, 128\n for North America)
\n
RSA encryption of bulk encryption keys and digital signatures
\n
SHA-1 as a secure hash algorithm
\n
\n\n
In the absence of digital signatures, Open Financial Exchange\napplications should utilize the HMAC keyed MAC algorithm, using SHA-1 as a secure hash\nfunction.
\n\n
When you set the tags for application-layer security-which determines\nwhether to use PKCS#7 message format-in the FI profile, the application software uses\nthese facilities. \n\n\n
Most of the content in Open Financial Exchange is language-neutral.\nHowever, some error messages, balance descriptions, and similar tags contain text meant to\nappear to the financial institution customers. There are also cases, such as e-mail\nrecords, where customers need to send text in other languages. To support world-wide\nlanguages, Open Financial Exchange must identify the basic text encoding, specific\ncharacter set, and the specific language.
\n\n
The outer Open Financial Exchange headers specify the encoding and\ncharacter set, as described Chapter 2. Current encoding values are ASCII and UNICODE. For\nASCII, character set values are code pages. Unicode ignores the character set per se\nalthough it still requires the syntax. Clients identify the language in the signon\nrequest. Open Financial Exchange specifies languages by three-letter codes as defined in\nISO-639. Servers report their supported languages in the profile (see Chapter 7). If a\nserver cannot support the requested language, they must return an error and not process\nthe rest of the transactions.\n\n
In each transaction involving amounts, responses include a default\ncurrency identification, <CURDEF>. The values are based on the ISO 4217 three-letter\ncurrency identifiers.
\n\n
Within each transaction, specific parts of the response might need to\nreport a different currency. Where appropriate, aggregates will include an optional\n<CURRENCY> aggregate. The scope of a <CURRENCY> aggregate is everything within\nthe same aggregate that the <CURRENCY> aggregate appears in, including nested\naggregates, unless overridden by a nested <CURRENCY> aggregate. For example,\nspecifying a <CURRENCY> aggregate in an investment statement detail means that the\nunit price, transaction total, commission, and all other amounts are in terms of the given\ncurrency, not the default currency.
\n\n
Note that there is no way for two or more individual elements that\nrepresent amounts-and are directly part of the same aggregate-to have different\ncurrencies. For example, there is no way in a statement download to have a different\ncurrency for the <LEDGERBAL> and the <AVAILBAL>, because they are both\ndirectly members of <STMTRS>. In most cases, you can use the optional <BAL>\nrecords to overcome this limitation, which do accept individual <CURRENCY>\naggregates.
\n\n
The default currency for a request is the currency of the source\naccount. For example, the currency for <BANKACCTFROM>.
\n\n
The <CURRATE> should be the one in effect throughout the scope of\nthe <CURRENCY> aggregate. It is not necessarily the current rate. Note that the\n<CURRATE> needs to take into account the choice of the FI for formatting of amounts\n(that is, where the decimal is) in both default and overriding currency, so that a client\ncan do math. This can mean that the rate is adjusted by orders of magnitude (up or down)\nfrom what is commonly reported in newspapers.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<CURRENCY>or\n \n <ORIGCURRENCY>
\n
Currency aggregate
\n
\n
\n
<CURSYM>
\n
ISO 4217 3-letter currency identifier, A-3
\n
\n
\n
<CURRATE>
\n
Ratio of <CURDEF> currency to <CURSYM>\n currency, in decimal form
\n
\n
\n
</CURRENCY>\n or \n </ORIGCURRENCY>
\n
\n
\n
\n\n
In some cases, Open Financial Exchange will define transaction responses so that\namounts have been converted to the home currency. However, Open Financial Exchange will\nallow FIs to optionally report the original amount and the original (foreign) currency. In\nthese cases, transactions include a specific tag for the original amount, and then a\n<ORIGCURRENCY> tag to report the details of the foreign currency.
\n\n
Again, <CURRENCY> means that Open Financial Exchange has not converted\namounts. Whereas, <ORIGCURRENCY> means that Open Financial Exchange has\nalready converted amounts.\n\n
Some of the tags in Open Financial Exchange have values that are\ncountry-specific. For example, <USPRODUCTTYPE> is only useful within the United\nStates. Open Financial Exchange will extend in each country as needed to provide tags that\naccept values useful to that country. Clients in other countries that do not know about\nthese tags will simply skip them.
\n\n
In some cases, a tag value represents a fundamental way of identifying\nsomething, yet there does not exist a world-wide standard for such identification.\nExamples include bank accounts and securities. In these cases, it is important that Open\nFinancial Exchange defines a single, extensible approach for identification. For example,\nCUSIPs are used within the U.S., but not in other countries. However, CUSIPs are\nfundamental to relating investment securities, holdings, and transactions. Thus, a\nsecurity ID consists of a two-part aggregate: one to identify the naming scheme, and one\nto provide a value. Open Financial Exchange will define valid naming schemes as necessary\nfor each country. \n \n \n\n\n
Currently, some systems provide only limited support for error recovery\nand no support for backup files or multiple clients. The Open Financial Exchange data\nsynchronization approach described in this chapter handles all of these situations.
\n\n
Open Financial Exchange defines a powerful form of data synchronization\nbetween clients and servers.
\n\n
Open Financial Exchange data synchronization addresses the following\nproblems:\n\n
\n
Error recovery
\n
Use of multiple client applications
\n
Restoring from a backup file
\n
Multiple data files (for example, one copy at home, another at work).\n
\n
\n\n
This chapter first provides a brief background of error recovery\nproblems and then presents the basic strategy used in Open Financial Exchange to perform\ndata synchronization. Each Open Financial Exchange service includes specific details for\ndata synchronization requests and responses.
\n\n
Most of the information in this chapter concerns data synchronization,\nsince it is a relatively new concept. The final section in this chapter discusses\nalternatives to full synchronization, and summarizes the options for each.\n\n
When a client begins a connection with a server for which the connection\ndoes not successfully complete, there are two main problems:\n\n
\n
Unconfirmed requests
\n
\n\n
If a client does not receive a response to work it initiates, it has no\nway of knowing whether the server processed the request. It also will not have any\nserver-supplied information about the request, such as a server ID number.\n\n
\n
Unsolicited data
\n
\n\n
Some banking protocols allow a server to send data to the client\nwhenever the client makes a connection. This specification assumes that the first client\nthat calls in after the unsolicited data is available for download receives the data. If\nthe connection fails, this information would be forever lost to the client. Examples of\nunsolicited data include updates in the status of a bill payment and e-mail responses.\n
\n\n
Unsolicited data presents problems beyond error recovery. Because the\nfirst client that connects to a server is the only one to receive unsolicited data, this\nsituation precludes use of multiple clients without a data synchronization method. For\nexample, if a user has a computer at work and one at home, and wants to perform online\nbanking from both computers, a bank server could send unsolicited data to one but not the\nother.
\n\n
An even greater problem occurs when a user resorts to a backup copy of\nthe client data file. This backup file will be missing recent unsolicited data with no way\nto retrieve it from the server once the server sends it.\n\n
A simple solution is to make sure that clients can always obtain\ninformation from the server for a reasonable length of time. Clients can request recent\nresponses-whether due to client-initiated work or other status changes on the server-by\nsupplying the previous endpoint in the response history. Servers always supply a new\nendpoint whenever they supply responses.
\n\n
If a client connection fails-or a client receives a response, but\ncrashes before updating its database-the client will not save the new endpoint. On the\nnext synchronization request, the server sends the same information (plus any further\nstatus changes).
\n\n
If a user switches to a backup file, again the client will use the older\nendpoint in the synchronization request.
\n\n
If multiple clients are in use, each will send requests based on its own\nendpoint, so that both clients will obtain complete information from the server. This is\none reason why Open Financial Exchange responses carry enough information from the request\nto enable them to be processed on their own. The diagram below shows the relationship\nbetween clients and servers. \n \n
\n\n
Open Financial Exchange relieves the server from maintaining any special\nerror-recovery state information. However, Open Financial Exchange requires the server to\nmaintain a history of individual responses it would have sent and a way to identify a\nposition in the history. This ID could be a timestamp, or be based on its existing state\ninformation.
\n\n
NOTE: Open Financial Exchange does not require servers to\nstore these responses based on individual connections. Also, not all requests are subject\nto synchronization. For example, Open Financial Exchange does not require servers to store\nstatement-download responses separately for data synchronization. \n\n
Open Financial Exchange does synchronization separately for each type of\nresponse. In addition, a synchronization request might include further identifying\ninformation, such as a specific account number. This specification defines the additional\ninformation for each synchronization request.
\n\n
Each Open Financial Exchange service will identify the specific\nresponses that are subject to data synchronization. For example, a bank statement-download\nis a read-only operation from the server. A client can request again if it fails;\nconsequently, there is no special data synchronization for this type of response.
\n\n
The basis for synchronization is a token as defined by the\n<TOKEN> tag. The server is free to create a token in any way it wishes. The client\nsimply holds the token for possible use in a future synchronization request.
\n\n
The server can derive a token from one of the following: \n\n
\n
Timestamp
\n
Sequential number
\n
Unique non-sequential number
\n
Other convenient values for a server
\n
\n\n\n\n
Clients will send a <TOKEN> of zero on their first synchronization\nrequest. Servers should send all available history, allowing a new client to know about\nwork done by other clients. If a user's account has never been used with Open Financial\nExchange, the server returns no history.
\n\n
The server can use different types of tokens for different types of\nresponses, if suitable for the server.
\n\n
Tokens will be subject to a maximum size; see Chapter 3, \"Common\nAggregates, Elements, and Data Types.\" Tokens need to be unique only with respect to\na specific type of synchronization request and the additional information in that request.\nFor example, a bill payment synchronization request takes an account number; therefore, a\ntoken needs to be unique only within payments for a specific account.
\n\n
Servers will not have infinite history available, so synchronization\nresponses include a <LOSTSYNC> element with a value of Y (yes) if the old token in\nthe synchronization request was older than available history. This tag allows clients to\nalert users that some responses have been lost.
\n\n
NOTE: A token is unrelated to a <TRNUID>,\n<SRVRTID>, or <FITID>. Each of these serves a specific purpose, and has its\nown scope and lifetime.
\n\n
A <SRVRTID> is not appropriate as a <TOKEN> for bill\npayment. A single payment has a single <SRVRTID>, but it can undergo several state\nchanges over its life and thus have several entries in the token history.
\n\n
There are three different ways a client and a server can conduct their\nrequests and responses:\n\n
\n
Explicit synchronization - A client can request synchronization without\n sending any (other) Open Financial Exchange requests. Clients will send a specific\n synchronization request, including the current token for that request. The response will\n be a set of individual responses more recent than the given token, along with a new token.\n
\n
Synchronization with new requests - A client can request synchronization\n as part of the response to any new requests it has. It gives the old token. The response\n should include responses to the new requests plus any others that became available since\n the old token, along with a new token. An aggregate contains the requests so that the\n server can process the new requests and update the token as an atomic action.
\n
New requests without synchronization - A client can make new requests\n without providing the old token. In this case, it expects just responses to the new\n requests. A subsequent request for synchronization will cause the client to send the same\n response again, because the client did not update its token.
\n
\n\n
Each request and response that requires data synchronization will define\na synchronization aggregate. The aggregate tells the server which particular kind of data\nit should synchronize. By convention, these aggregates always have SYNC as part of their\ntag names, for example, <PMTSYNCRQ>. You can use these aggregates on their\nown to perform explicit synchronization, or as wrappers around one or more new\ntransactions. For example, you can use <PMTSYNCRQ> aggregates to request\nsynchronization in combination with new work. You can use the <PMTTRNRQ> by itself\nif you do not require synchronization.
\n\n
Some clients can choose to perform an explicit synchronization before\nsending any new requests (with or without synchronization). This practice allows clients\nto be up-to-date and possibly interact with the user before sending any new requests.\nOther clients can simply send new requests as part of the synchronization request.
\n\n
If a client synchronizes in one file, then sends new work inside a\nsynchronization request in a second file, there is a small chance that additional\nresponses become available between the two connections. There is even a smaller chance\nthat these would be conflicting requests, such as modifications to the same object.\nHowever, some clients and some requests might require absolute control, so that the user\ncan be certain that they are changing known data. To support this, synchronization\nrequests can optionally specify <REJECTIFMISSING>. The tag tells a server that it\nshould reject all enclosed requests if the supplied <TOKEN> is out of date before\nconsidering the new requests. That is, if any new responses became available, whether\nrelated to the incoming requests or not (but part of the scope of the synchronization\nrequest), the server should immediately reject the requests. It should still return the\nnew responses. A client can then try again until it finds a stable window to submit the\nwork. See section 6.5 for more information about conflict detection and resolution.\n
\n\n
The password change request and response present a special problem. See\nsection 2.5.2 for further information.\n\n
Conflicts arise whenever two or more users or servers modify the same\ndata. This can happen to any object that has a <SRVRTID> that supports change or\ndelete requests. For example, one spouse and the other might independently modify the same\nrecurring bill payment model. From a server perspective, there is usually no way to\ndistinguish between the same user making two intended changes and two separate users\nmaking perhaps unintended changes. Therefore, Open Financial Exchange provides enough\ntools to allow clients to carefully detect and resolve conflicts. Open Financial Exchange\nrequires only that a server process atomically all requests in a single <OFX> block.\n
\n\n
A careful client will always synchronize before sending any new\nrequests. If any responses come back that could affect a user's pending requests, the\nclient can ask the user whether it should still send those pending requests. Because there\nis a small chance for additional server actions to occur between the initial\nsynchronization request and sending the user's pending requests, extremely careful clients\ncan use the <REJECTIFMISSING> option. Clients can iterate sending pending requests\ninside a synchronization request with <REJECTIFMISSING> and testing the responses to\nsee if they conflict with pending requests. A client can continue to do this until a\nwindow of time exists wherein the client is the only agent trying to modify the server. In\nreality, this will almost always succeed on the first try.\n\n
There are some situations, and some types of clients, where it is\npreferable for a client to ask a server to send everything it knows, rather than just\nreceive a set of changes. For example, a situation where a client that has not connected\noften enough has lost synchronization. An example of \"type\" might be a\ncompletely stateless client, such as a web browser. This choice is made during client\nimplementation. Open Financial Exchange does not require a client to refresh just because\nit has lost synchronization.
\n\n
Clients will mainly want to refresh lists of long-lived objects on the\nserver; generally objects with a <SRVRTID>. For example, Open Financial Exchange\nPayments has both individual payments and models of recurring payments.
\n\n
A brand new client, or a client that lost synchronization, might want to\nlearn about in-progress payments by doing a synchronization refresh of the payment\nrequests. It would almost certainly want to do a synchronization refresh of the recurring\npayment models, because these often live for months or years.
\n\n
A client might not perform a synchronization refresh on e-mail\nresponses.
\n\n
A client can request a refresh by using the <REFRESH> tag with\nvalue of Y instead of the <TOKEN> tag. Server descriptions detail the exact behavior\nthat servers should follow. However, the general rule is that servers should send\nresponses that emulate a client creating or adding each of the objects governed by the\nparticular synchronization request.
\n\n
In these cases, you can set <TRNUID> to zero; the standard value\nfor server-generated responses.
\n\n
There is no need to recreate a stream of responses that emulate the\nentire history of the object, just an add response that reflects the current state. For\nexample, if you create a model and then modify it three times, even if this history would\nhave been available for a regular synchronization, servers should only send a single add\nthat reflects the current state.
\n\n
A client that just wants the current token, without refresh or\nsynchronization, makes requests with <TOKENONLY> and a value of Y.
\n\n
In all cases, servers should send the current ending <TOKEN> for\nthe synchronization request in refresh responses. This allows a client to perform regular\nsynchronization requests in the future.
\n\n
The following table summarizes the options in a client synchronization\nrequest:
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<TOKEN>
\n
Previous value of <TOKEN> received for this type of\n synchronization request from server; 0 for first-time requests; token
\n
\n
\n
<TOKENONLY>
\n
Request for just the current <TOKEN> without the\n history, Boolean
\n
\n
\n
<REFRESH>
\n
Request for refresh of current state, Boolean\n
\n
\n
\n
<REJECTIFMISSING>
\n
If Y, do not process requests if client <TOKEN> is\n out of date, Boolean
\n
\n
\n\n
NOTE:Open Financial Exchange requires one each of <TOKEN>,\n<TOKENONLY>, or <REFRESH>.\n\n
This section describes how an FI can approach supporting synchronization\nbased on the assumption that modifications to an existing financial server will be kept to\na minimum.
\n\n
The simplest approach is to create a history database separate from the\nexisting server. This history could consist of the actual Open Financial Exchange\ntransaction responses (<TRNRS> aggregates) that are available to a synchronization\nrequest. The history database could index records by token, response type, and any other\nidentifying information for that type, such as account number.
\n\n
The diagram below shows a high-level model of the Open Financial\nExchange architecture for a financial institution. Notice that the diagram shows the\npresence of a history journal. \n \n
\n\n
The server adds responses to the history journal for any action that\ntakes place on the existing server. This is true whether the Open Financial Exchange\nrequests initiate the action or, in the case of recurring payments, it happens\nautomatically on the server. Once added to the history journal, the server can forget\nthem.
\n\n
The areas of the Open Financial Exchange server that process\nsynchronization requests need only search this history database for matching responses\nthat are more recent than the incoming token.
\n\n
For a refresh request, an Open Financial Exchange server would access\nthe actual bank server to obtain the current state rather than recent history.
\n\n
Periodically the bank server would purge the history server of older\nentries.
\n\n
Only requests that are subject to synchronization need to have entries\nin the history database. Statement downloads do not involve synchronization; therefore,\nthe FI server should not add these responses to the history database. Since statement\ndownloads are usually the largest in space and the most frequent, eliminating these saves\nmuch of the space a response history might otherwise require.
\n\n
More sophisticated implementations can save even more space. The history\ndatabase could save responses in a coded binary form that is more compact than the full\nOpen Financial Exchange response format. Some FIs might have much or all of the necessary\ndata already in their servers; consequently, they would not require new data. An FI could\nregenerate synchronization responses rather than recall them from a database.\n\n
The diagram below shows a general flowchart of what an Open Financial\nExchange client would do with the results of a synchronization request. Most requests and\nresponses subject to data synchronization contain both <TRNUID> and <SRVRTID>.\n \n \n \n\n\n
It is increasingly common that a server can get simultaneous or\noverlapping requests from the same user over two different computers. Open Financial\nExchange requires a server to process each set of requests sent in a file as an atomic\naction. Servers can deal with the problems that arise with simultaneous use in two ways:\n\n
\n
Allow simultaneous connections, insure each is processed atomically, and\n use the data synchronization mechanism to bring the two clients up to date. This is the\n preferred method.
\n
Lock out all but one user at a time, returning the error code for\n multiple users.
Although it is RECOMMENDED that Open Financial Exchange servers\nimplement full synchronization as described in this chapter, an alternate approach,\n\"lite synchronization,\" could be easier for some servers to support. This\napproach focuses only on error recovery and does not provide any support for multiple\nclients, multiple data files, or use of backup files. The approach is to preserve the\nmessage sets while simplifying the implementation.
\n\n
In addition, some clients might prefer to use response-file based error\nrecovery with all servers, even if the client and some server both support full\nsynchronization. This section first describes lite synchronization, and then explains the\nrules that clients and servers use to decide how to communicate.\n\n
Lite synchronization requires servers to accept all synchronization\nmessages, but does not require them to keep any history or tokens. Responses need only be\nsent once and then the server can forget them. Responses to client requests, whether or\nnot they are made inside a synchronization request, are processed normally. Responses that\nrepresent server-initiated work, such as payment responses that arise from recurring\npayments, are sent only in response to synchronization requests. A server does not have to\nhold responses in case a second client makes a synchronization request.
\n\n
Because full synchronization supports error recovery, an alternative is\nneeded for lite synchronization. Servers using lite synchronization keep a copy of the\nentire response file they last sent. Clients requesting that servers prepare for error\nrecovery generate a globally unique ID for each file they send. In the OFX headers, there\nare two tags associated with error recovery:\n\n
\n
OLDFILEUID - UID of the last request and response that was successfully\n received and processed by the client
\n
NEWFILEUID - UID of the current file
\n
\n\n
The format of these is the same as used with <TRNUID> as\ndocumented in Chapter 3.
\n\n
Servers use the following rules:\n\n
\n
If these tags are absent, the client is not requesting error recovery\n protection for this file. The server does not need to save a copy of the response.
\n
If the NEWFILEUID matches a file saved on the server, then the client is\n in error recovery. The server must ignore the new requests in this file and instead send\n back the matching saved response file.
\n
If the OLDFILEUID matches a file saved on the server, then OLDFILEUID is\n a file that the client has successfully processed and the server can delete it. The client\n is also requesting that the response for the current file be saved under the NEWFILEUID\n for possible error recovery.
\n
\n\n
A server will never need to save more than one file per client data\nfile, but because of possible multi-client or multi-datafile usage, it might need to save\nseveral files for a given user. A server should save as long as possible, but not\nindefinitely. A server cannot recognize an error recovery attempt if it comes after it has\npurged the error recovery file. A server would process it as a new request. In this case,\na server should recognize duplicate transaction UIDs for client-initiated work, such as\npayments, and then reject them individually. Server-generated responses would be lost to\nthe client.
\n\n
For a server accustomed to sending unsolicited responses, lite\nsynchronization should closely match the current response-file based implementation. The\nonly difference is that a server should hold the unsolicited responses until the client\nmakes the first appropriate synchronization request; rather than automatically adding them\nto any response file. Once added, the server can mark them as delivered, relying on error\nrecovery to insure actual delivery.\n\n
Client and server developers should first decide whether they will\nsupport full synchronization or not. If they can, then they can support response-file\nerror recovery as well, or they can rely on synchronization to perform error recovery. If\nthey adopt only lite synchronization, Open Financial Exchange requires response-file error\nrecovery. A severs describes each of these choices in its server profile records. The\nfollowing combinations are valid:\n\n
\n
Full synchronization with response-file error recovery
\n
Full synchronization without separate response-file error recovery\n
\n
Lite synchronization with response-file error recovery
\n
\n\n
Clients request response-file error recovery by including the old and\nnew session UIDs in the header. If they are absent, servers need not save the response\nfile for error recovery. Clients request synchronization by using those synchronization\nrequests defined throughout this specification.\n\n
Here is an example of full synchronization using bill payment as the\nservice. Open Financial Exchange Payments provides two different synchronization requests\nand responses, each with their own token; one for payment requests and one for repeating\npayment model requests. See Chapter 102 for full details.
\n\n
These simplified examples, show without the outer <OFX> layer,\n<SIGNON>, and so forth.Client A requests synchronization:
\n\n
<PMTSYNCRQ>\n\t<TOKEN>123\n\t<BANKACCTFROM>\n\t\t<BANKID>121000248\n\t\t<ACCTID>123456789\n\t\t<ACCTTYPE>CHECKING\n\t</BANKACCTFROM>\n</PMTSYNCRQ>The server sends in response:\n<PMTSYNCRS>\n\t<TOKEN>125\n\t<LOSTSYNC>N\n\t<BANKACCTFROM>\n\t\t<BANKID>121000248\n\t\t<ACCTID>123456789\n\t\t<ACCTTYPE>CHECKING\n\t</BANKACCTFROM>\n\t<PMTTRNRS>\n\t\t<STATUS>\n\t\t\t... status details\n\t\t</STATUS>\n\t\t<TRNUID>123\n\t\t<PMTRS>\n\t\t\t... details on a payment response\n\t\t</PMTRS>\n\t</PMTTRNRS>\n\t<PMTTRNRS>\n\t\t<STATUS>\n\t\t\t... status details\n\t\t</STATUS>\n\t\t<TRNUID>546\n\t\t<PMTRS>\n\t\t\t... details on another payment response\n\t\t</PMTRS>\n\t</PMTTRNRS>\n</PMTSYNCRS>\n
\n\n
Client A was missing two payment responses, which the server provides.\nAt this point, client A is synchronized with the server. Client A now makes a new payment\nrequest, and includes a synchronization update as part of the request. This update avoids\nhaving to re-synchronize the expected response at a later time.
\n\n
<PMTSYNCRQ>\n\t<TOKEN>125\n\t<BANKACCTFROM>\n\t\t<BANKID>121000248\n\t\t<ACCTID>123456789\n\t\t<ACCTTYPE>CHECKING\n\t</BANKACCTFROM>\n\t<PMTTRNRQ>\n\t\t<TRNUID>12345\n\t\t<PMTRQ>\n\t\t\t... details of a new payment request\n\t\t</PMTRQ>\n\t</PMTTRNRQ>\n</PMTSYNCRQ>The response to this new\nrequest:\n<PMTSYNCRS>\n\t<TOKEN>126\n\t<LOSTSYNC>N\n\t<BANKACCTFROM>\n\t\t<BANKID>121000248\n\t\t<ACCTID>123456789\n\t\t<ACCTTYPE>CHECKING\n\t</BANKACCTFROM>\n\t<PMTTRNRS>\n\t\t... details on a payment response to the new request\n\t</PMTTRNRS>\n</PMTSYNCRS>\n
\n\n
The client now knows that the server has processed the payments request\nit just made, and that nothing else has happened on the server since it last synchronized\nwith the server.
\n\n
Assume client B was synchronized with respect to payments for this\naccount up through token 125. If it called in now and synchronized-with or without making\nadditional requests-it would pick up the payment response associated with token 126. It\nrecords the same information that was in client A, which would give both clients a\ncomplete picture of payment status. \n\n\n
Open Financial Exchange clients use the profile to learn the\ncapabilities of an Open Financial Exchange server. This information includes general\nproperties such as account types supported, user password requirements, specific messages\nsupported, and how the client should batch requests and where to send the requests. A\nclient obtains a portion of the profile when a user first selects an FI. The client\nobtains the remaining information prior to sending any actual requests to that FI. The\nserver uses a timestamp to indicate whether the server has updated the profile, and the\nclient checks periodically to see if it should obtain a new profile.
\n\n
In more detail, a profile response contains the following sections,\nwhich a client can request independently:\n\n
\n
Message Sets - list of services and any general attributes of those\n services. Message sets are collections of messages that are related functionally. They are\n generally subsets of what users see as a service.
\n
Signon realms - FIs can require different signons (user ID and/or\n password) for different message sets. Because there can only be one signon per <OFX>\n block, a client needs to know which signon the server requires and then provide the right\n signon for the right batch of messages.
\n
\n\n
The profile message is itself a message set. In files, Open Financial\nExchange uses the <PROFMSGSV1> aggregate to identify this profile message set.\n
\n\n
The following sections describe the general use of profile information. \n\n
A message set is a collection of related messages. For example, Chapter\n11, \"Banking,\" defines several message sets: statement download, credit card\nstatement download, intrabank transfers, and so forth. A server routes all of the messages\nin a message set to a single URL and merges their versions together.
\n\n
Clients and servers generally use message sets as the granularity to\ndecide what functionality they will support. A \"banking\" server can choose to\nsupport the statement download and intrabank transfer message sets, but not the wire\ntransfer message set. Attributes are available in many cases to further define how Open\nFinancial Exchange supports a message set.
\n\n
Each portion of the Open Financial Exchange specification that defines\nmessages also defines the message set to which that the messages belongs. This includes\nwhat additional attributes are available for those messages, and whether Open Financial\nExchange requires the message set or it is optional.\n\n
Message sets are the basis of version control. Over time there will be\nnew versions of the message sets, and at any given time servers will likely want to\nsupport more than one version of a message set. Clients should also be capable of\nsupporting as many versions as possible. Through the profile, clients discover which\nversions are supported for each message set. Considering the client capabilities, it\nexchanges messages at the highest common level for each message set.
\n\n
For the Open Financial Exchange-SGML data format, there is a single DTD\nfor all message sets. Its version advances when any syntactic change is made to any\nof the message sets. (It is possible to make a semantic change that would not even\nrequire a change in syntax. A change in rules, for example, that would change the version\nof the message set without changing the DTD.) A single DTD cannot have two different\ndefinitions for the same aggregate. There are limitations to how a server that uses true\nDTD-based parsing can handle multiple versions of a message at the same time.\n\n
To allow FIs to set up different servers for different message sets,\ndifferent versions, or to directly route some messages to third party processors, message\nsets define the URL to which a server sends messages in that message set. Each version of\na message set can have a different URL. In the common case where many or all message sets\nare sent to a single URL, clients will consolidate messages across compatible message\nsets. Clients can consolidate when:\n\n
A profile request indicates which profile components a client desires.\nIt also indicates what the client's routing capability is. Profiles returned by the FI\nmust be compatible with the requested routing style, or it returns an error.
\n\n
Profile requests are not subject to synchronization. Use the\n<PROFTRNRQ> transaction tag.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<PROFRQ>
\n
Profile-request aggregate
\n
\n
\n
<CLIENTROUTING>
\n
Identifies client routing capabilities, see table below\n
\n
\n
\n
<DTPROFUP>
\n
Date and time client last received a profile update
\n
\n
\n
</PROFRQ>
\n
\n
\n
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
NONE
\n
Client cannot perform any routing. All URLs must be the\n same. All message sets share a single signon realm.
\n
\n
\n
SERVICE
\n
Client can perform limited routing. See details below.\n
\n
\n
\n
MSGSET
\n
Client can route at the message-set level. Each message set\n may have a different URL and/or signon realm.
\n
\n
\n\n
The intent of the SERVICE option for client routing is to support clients that can\nroute bill payment messages to a separate URL from the rest of the messages. Because the\nexact mapping of message sets to the general concept of bill payment can vary by client\nand by locale, this specification does not provide precise rules for the SERVICE option.\nEach client will define its requirements. \n\n
An aggregate describes each message set supported by an FI. Message sets\nin turn contain an aggregate for each version of the message set that is supported. For a\nmessage set named XXX, the convention is to name the outer aggregate <XXXMSGSET>\nand the tag for each version <XXXMSGSETVn>. The reason for message\nset-specific aggregates is that the set of attributes depends on the message set. These\ncan change from version to version, so there are version-specific aggregates as well.\n
\n\n
The general form of the response is:
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<XXXMSGSET>
\n
Service aggregate
\n
\n
\n
<XXXMSGSETVn>
\n
Version-of-message-set aggregate, 1 or more
\n
\n
\n
</XXXMSGSETVn>
\n
\n
\n
\n
</XXXMSGSET>
\n
\n
\n
\n\n
The <XXXMSGSETVn> aggregate has the following form: \n
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<XXXMSGSETVn>
\n
Message-set-version aggregate
\n
\n
\n
<MSGSETCORE>
\n
Common message set information
\n
\n
\n
</MSGSETCORE>
\n
\n
\n
\n
message-set specific
\n
Zero or more attributes specific to this version of this\n message set, as defined by each message set
\n
\n
\n
</XXXMSGSETVn>
\n
\n
\n
\n\n
The common message set information <MSGSETCORE> is as follows: \n
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<MSGSETCORE>
\n
Common-message-set-information aggregate
\n
\n
\n
<VER>
\n
Version number, N-5 (version 1.0 formatted as 100)\n
\n
\n
\n
<URL>
\n
URL where messages in this set are to be sent
\n
\n
\n
<OFXSEC>
\n
Security level required for this message set; see Chapter 4\n
\n
\n
\n
<TRANSPSEC>
\n
Y if transport security must be used, N if not used; Boolean\n
A signon realm identifies a set of messages that can be accessed using\nthe same password. Realms are used to disassociate signons from specific services,\nallowing FIs to require different signons for different message sets. In practice, FIs\nwill want to use the absolute minimum number of realms possible to reduce the user's\nworkload.
The profile message set functions the same way as all other message\nsets; therefore, it contains a profile description for that message set. Because\n<PROFMSGSET> is always part of a message set response, it is described here. Servers\nthat support profile information must include the <PROFMSGSET> as part of the\nprofile response <MSGSETLIST>. There are no attributes, but the aggregate must be\npresent to indicate support for the message set. \n
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<PROFMSGSET>
\n
Message-set-profile-information aggregate
\n
\n
\n
<PROFMSGSETV1>
\n
Opening tag for V1 of the message set profile information\n
The Signup message set defines three messages to help users get setup\nwith their FI:\n\n
\n
Enrollment - informs FI that a user wants to use Open Financial Exchange\n and requests that a password be returned
\n
Accounts - asks the FI to return a list of accounts, and the services\n supported for each account
\n
Activation - allows a client to tell the FI which services a user wants\n on each account
\n
\n\n
There is also a message to request name and address changes.
\n\n
Clients use the account information request on a regular basis to look\nfor changes in a user's account information. A timestamp is part of the request so that a\nserver has only to report new changes. Account activation requests are subject to data\nsynchronization, and will allow multiple clients to learn how the other clients have been\nenabled.
\n\n
In Open Financial Exchange files, the <SIGNUPMSGSV1> aggregate\nidentifies the Signup message.\n\n
The message sets in this chapter are designed to allow both FIs and\nclients to support a variety of sign-up procedures. There are four basic steps a user\nneeds to go through to complete the sign-up:\n\n
\n
Select the FI. Open Financial Exchange does not define this step\n or provide message sets to support it. Client developers and FIs can let a user browse or\n search this information on a web site, or might define additional message sets to do this\n within the client. At the conclusion of this step, the client will have some minimal\n profile information about the FI, including the set of services supported and the URL to\n use for the next step.
\n
Enrollment and password acquisition. In this step, the user\n identifies and authenticates itself to the FI without a password. In return, the\n user obtains a password (possibly temporary) to use with Open Financial Exchange. FIs can\n perform this entire step over the telephone, through a combination of telephone requests\n and a mailed response, or at the FI web site. FIs can also use the Open Financial Exchange\n enrollment message to do this by means of the client. The response can contain a temporary\n password or users can wait for a mailed welcome letter containing the password.
\n
Account Information. In this step, the user obtains a list of\n accounts available for use with Open Financial Exchange, and which specific services are\n available for each account. Even if users have enrolled over the telephone, clients will\n still use this message set to help users properly set up the accounts within the client.\n Clients periodically check back with the FI for updates.
\n
Service Activation. The last step is to activate specific services\n on specific accounts. The activation messages support this step. Synchronization is\n applied to these messages to insure that other clients are aware of activated services.\n
\n\n\n
The combination of media-interface through which an FI accomplishes\nthese steps can vary. FIs might wish to do steps two through four over the telephone.\nClients will still use Open Financial Exchange messages in steps 3 and 4 to automatically\nset up the client based on the choices made by the user over the phone. Other FIs might\nwish to have the entire user experience occur within the client. Either way, the Open\nFinancial Exchange sign-up messages support the process.\n\n
To support the widest possible set of FIs, Open Financial Exchange\nassumes that individual users and accounts are in a many-to-many relationship. Consider a\nhousehold with three accounts:\n\n
\n
Checking 1 - held individually by one spouse
\n
Checking 2 - held jointly by both
\n
Checking 3 - held individually by the other spouse
\n
\n\n
Checking 2 should be available to either spouse, and the spouse holding\nChecking 1 should be able to see both Checking 1 and 2.
\n\n
Open Financial Exchange expects FIs to give each user their own user ID\nand password. Each user will go through the enrollment step separately. A given account\nneed only be activated once for a service; not once for each user. Clients will use the\naccount information and activation messages to combine information about jointly-held\naccounts.
\n\n
If an FI prefers to have a single user ID and password per household or\nper master account, they will have to make this clear to users through the enrollment\nprocess. It is up to the FI to assign a single user ID and password that can access all\nthree of the checking accounts described above.\n\n
The main purpose of the enrollment message is to communicate a user's\nintent to access the FI by way of Open Financial Exchange and to acquire a password for\nfuture use with Open Financial Exchange. Some FIs might return a user ID and an initial\npassword in the enrollment response, while others will send them by way of regular mail.
\n\n
NOTE: Because the server does not know the user ID and\npassword when the client sends the enrollment request, the <SONRQ> will not contain\na valid user ID or password. The enrollment message accepts standard user identification\ninformation.
\n\n
Enrollment requests are not subject to synchronization. If the client\ndoes not receive a response, it will simply re-request the enrollment. If a user\nsuccessfully enrolls from another client before the first client obtains a response, the\nserver should respond to subsequent requests from the first client with status code:
The Open Financial Exchange <SONRQ> requires a user ID to uniquely\nidentify a user to an FI. Many FIs in the United States use social security numbers (SSNs)\nas the ID. Others create IDs that are unrelated to the users' SSNs. FIs can have an\nexisting user IDs that they use for other online activities that they wish to use for Open\nFinancial Exchange as well. They might also create new IDs specifically for Open Financial\nExchange. Finally, some FIs might assign IDs while others might allow users to create\nthem.
\n\n
Because users do not usually know either their Open Financial Exchange\nsign-on user ID or their password at time of enrollment, the enrollment response is\ndesigned to return both. The enrollment request allows users to optionally provide a user\nID, which an FI can interpret as their existing online ID or a suggestion for what their\nnew user ID should be. It is recommended that the enrollment process explains ID syntax to\nusers.\n\n
The enrollment request captures enough information to identify and\nauthenticate a user as being legitimate and that it has a relationship with the FI.
\n\n
FIs might require that an account number be entered as part of the\nidentification process. However, this is discouraged since the account information request\nis designed to automatically obtain all account information, avoiding the effort and\npotential mistakes of a user-supplied account number.
\n\n
It is RECOMMENDED that FIs provide detailed specifications for\nIDs and passwords along with information about the services available when a user is\nchoosing an FI.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<ENROLLRQ>
\n
Enrollment-request aggregate
\n
\n
\n
<FIRSTNAME>
\n
First name of user
\n
\n
\n
<MIDDLENAME>
\n
Middle name of user
\n
\n
\n
<LASTNAME>
\n
Last name of user
\n
\n
\n
<ADDR1>
\n
Address line 1
\n
\n
\n
<ADDR2>
\n
Address line 2
\n
\n
\n
<ADDR3>
\n
Address line 3
\n
\n
\n
<CITY>
\n
City
\n
\n
\n
<STATE>
\n
State or province
\n
\n
\n
<POSTALCODE>
\n
Postal code
\n
\n
\n
<COUNTRY>
\n
3-letter country code from ISO/DIS-3166
\n
\n
\n
<DAYPHONE>
\n
Daytime telephone number
\n
\n
\n
<EVEPHONE>
\n
Evening telephone number
\n
\n
\n
<EMAIL>
\n
Electronic e-mail address
\n
\n
\n
<USERID>
\n
Actual user ID if already known, or preferred user ID if\n user can pick
\n
\n
\n
<TAXID>
\n
ID used for tax purposes (such as SSN), may be same as user\n ID
\n
\n
\n
<SECURITYNAME>
\n
Mother's maiden name or equivalent
\n
\n
\n
<DATEBIRTH>
\n
Date of birth
\n
\n
\n
<ACCTFROM>
\n
An account description aggregate for one existing account\n at the FI, for identification purposes only. Can be <BANKACCTFROM>,\n <INVACCTFROM>, etc.
\n
\n
\n
</ACCTFROM>
\n
\n
\n
\n
</ENROLLRQ>
\n
\n
\n
\n\n
This enrollment request is intended for use only by individuals. Business enrollment\nwill be defined in a later release.\n\n
The main purpose of the enrollment response is to acknowledge the\nrequest. In those cases where FIs permit delivery of an ID and a temporary password, the\nresponse also provides for this. Otherwise the server will send the real response to the\nuser by way of regular mail, electronic mail, or over the telephone. If enrollment is\nsuccessful, but the server does not return the ID and password in the response, a server\nis REQUIRED to use status code 10 and provide some information to the user by means of the\n<MESSAGE> element in the <STATUS> aggregate about what to expect next.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<ENROLLRS>
\n
Enrollment-response aggregate
\n
\n
\n
<TEMPPASS>
\n
Temporary password
\n
\n
\n
<USERID>
\n
User ID
\n
\n
\n
<DTEXPIRE>
\n
Time the temporary password expires (if <TEMPPASS>\n included)
<ENROLLTRNRQ>\n\t<TRNUID>12345\n\t<ENROLLRQ>\n\t\t<FIRSTNAME>Joe\n\t\t<MIDDLENAME>Lee\n\t\t<LASTNAME>Smith\n\t\t<ADDR1>21 Main St.\n\t\t<CITY>Anytown\n\t\t<STATE>TX\n\t\t<POSTALCODE>87321\n\t\t<COUNTRY>USA\n\t\t<DAYPHONE>123-456-7890\n\t\t<EVEPHONE>987-654-3210\n\t\t<EMAIL>jsmith@isp.com\n\t\t<USERID>jls\n\t\t<TAXID>123-456-1234\n\t\t<SECURITYNAME>jbmam\n\t\t<DATEBIRTH>19530202\n\t</ENROLLRQ>\n</ENROLLTRNRQ>And the reply might be:\n<ENROLLTRNRS>\n\t<TRNUID>12345\n\t<STATUS>\n\t\t<CODE>0\n\t\t<SEVERITY>INFO\n\t</STATUS>\n\t<ENROLLRS>\n\t\t<TEMPPASS>changeme\n\t\t<USERID>jls\n\t\t<DTEXPIRE>19970105\n\t</ENROLLRS>\n</ENROLLTRNRS>\n
Account information requests ask a server to identify and describe all\nof the accounts accessible by the signed-on user. The definition of all is up to\nthe FI. At a minimum, it is RECOMMENDED that a server include information about all\naccounts that it can activate for one or more Open Financial Exchange services. To give\nthe user a complete picture of his relationship with an FI, FIs can give information on\nother accounts, even if those accounts are available only for limited Open Financial\nExchange services.
\n\n
Some service providers will not have any prior knowledge of any user\naccount information. The profile allows these servers to report this, and clients will\nthen know to ask users for account information rather than reading it from the server.\n
\n\n
Clients can perform several tasks for users with this account\ninformation. First, the information helps a client set up a user for online services by\ngiving it a precise list of its account information and available services for each.\nClients can set up their own internal state as well as prepare service activation requests\nwith no further typing by users. This can eliminate data entry mistakes in account\nnumbers, routing transit numbers, and so forth.
\n\n
Second, FIs can provide limited information on accounts that would not\nordinarily be suitable to Open Financial Exchange services. For example, a balance-only\nstatement download would be useful for certificates of deposits even though a customer or\nan FI might not want or allow CDs to be used for full statement download.
\n\n
For each account, there is one <ACCTINFO> aggregate returned. The\naggregate includes one service-specific account information aggregate for each available\nservice on that account. That, in turn, provides the service-specific account\nidentification. Common to each service-specific account information aggregate is the\n<SVCSTATUS> tag, which indicates the status of this service on this account.
\n\n
A server should return joint accounts (accounts for which more than one\nuser ID can be used to access the account) for either user. Clients that wish to have a\nunified view will aggregate the results and remove duplicates before making specific\nrequests involving joint accounts.
\n\n
Requests and responses include a <DTACCTUP> element. Responses\ncontain the last time a server updated the information. Clients can OPTIONALLY send\nthis in a subsequent request, and servers are REQUIRED to compare this to the\ncurrent modification time and only send information if it is more recent. The server sends\nthe entire account information response if the client's time is older; there is no attempt\nto incrementally update specific account information.\n\n
URL to request the logo for the account (actual logos\n should be included via multi-part MIME in the response file if requested), URL\n
\n
\n
\n
<XXXACCTINFO>
\n
Service-specific account information, defined in each\n service chapter, one or more allowed
\n
\n
\n
<XXXACCTFROM>
\n
Service-specific account identification
\n
\n
\n
</XXXACCTFROM>
\n
\n
\n
\n
<SVCSTATUS>
\n
AVAIL = Available, but not yet requested
PEND = Requested, but not yet available
\n
ACTIVE = In use
\n
\n
\n
</XXXACCTINFO>
\n
\n
\n
\n
</ACCTINFO>
\n
\n
\n
\n\n
NOTE: A server uses the <DESC> field to convey the FI's preferred name\nfor the account, such as \"PowerChecking.\" It should not include the account\nnumber. \n\n
Clients inform FIs that they wish to start, modify, or terminate a\nservice for an account by sending service activation requests. These are subject to data\nsynchronization, and servers should send responses to inform clients of any changes, even\nif the changes originated on the server.
\n\n
Clients use these records during the initial user sign-up process. Once\na client learns about the available accounts and services (by using the account\ninformation request above, or by having a user directly enter the required information),\nit sends a series of service ADD requests.
\n\n
If a user changes any of the identifying information about an account,\nthe client sends a service activation request containing both the old and the new account\ninformation. Servers should interpret this as a change in the account, not a request to\ntransfer the service between two existing accounts, and all account-based information such\nas synchronization tokens should continue. If a user or FI is reporting that service\nshould be moved between two existing accounts, service must be terminated for the old\naccount and started for the new account. The new account will have reset token histories,\nas with any new service.
\n\n
Each service to be added, changed, or removed is contained in its own\nrequest because the same real-world account might require different <ACCTFROM>\naggregates depending on the type of service.\n\n
Service activation requests are subject to the standard data\nsynchronization protocol. The scope of these requests and the <TOKEN> is the\nuser-ID. The request and response tags are <ACCTSYNCRQ> and <ACCTSYNCRS>.\n\n
Users may request that an FI update the official name, address, phone,\nand e-mail information using the <CHGUSERINFORQ>. Only the fields that should be\nchanged are sent. The response reports all of the current values. For security reasons,\nsome of the fields in the <ENROLLRQ> cannot be changed online, such as tax ID.
\n\n
The transaction tag is <CHGUSERINFOTRNRQ> and\n<CHGUSERINFOTRNRSRQ>. These methods are subject to synchronization,\n<CHGUSERINFOSYNCRQ> and <CHGUSERINFOSYNCRS>.\n\n
A server must include the following aggregates as part of the profile\n<MSGSETLIST> response, since every server must support at least the account\ninformation and service activation messages. In the <ENROLLPROF> aggregate, servers\nindicate how enrollment should proceed: via the client, a given web page, or a text\nmessage directing users to some other method (such as a phone call)..
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<SIGNUPMSGSET>
\n
Signup-message-set-profile-information aggregate
\n
\n
\n
<SIGNUPMSGSETV1>
\n
Opening tag for V1 of the message set profile information\n
\n
\n
\n
<MSGSETCORE>
\n
Common message set information, defined in the profile\n chapter
\n
\n
\n
</MSGSETCORE>
\n
\n
\n
\n
\n
Enrollment options - only one of <CLIENTENROLL>,\n <WEBENROLL>, or <OTHERENROLL> is allowed
\n
\n
\n
<CLIENTENROLL>\n
\n
Client-based enrollment supported
\n
\n
\n
<ACCTREQUIRED>
\n
Y if account number is required as part of enrollment Boolean\n
\n
\n
\n
</CLIENTENROLL>\n
\n
\n
\n
\n
<WEBENROLL>\n
\n
Web-based enrollment supported
\n
\n
\n
<URL>
\n
URL to start enrollment process
\n
\n
\n
</WEBENROLL>
\n
\n
\n
\n
<OTHERENROLL>\n
\n
Some other enrollment process
\n
\n
\n
<MESSAGE>
\n
Message to give to consumer about what to do next (e.g. a\n phone number) A-80
\n
\n
\n
</OTHERENROLL>\n
\n
\n
\n
\n
<CHGUSERINFO>
\n
Y if server supports client-based user information changes\n
\n
\n
\n
<AVAILACCTS>
\n
Y if server can provide information on accounts with\n SVCSTATUS available, N means client should expect to ask user for specific account\n information Boolean
The e-mail message set includes two messages: generic e-mail and generic\nMIME requests by way of URLs. In Open Financial Exchange files, the message set name is\nEMAILMSGSV1.\n\n
Open Financial Exchange allows consumers and FIs to exchange messages.\nThe message body is in HTML so that FIs can provide some graphic structure to the message.\nKeep in mind that, as with regular World Wide Web browsing, an Open Financial Exchange\nclient might not support some or all of the HTML formatting, so the text of the message\nmust be clear on its own. Clients can request that graphics (the images referenced in an\n<IMG> tag) be sent as part of the response file, or clients can separately request\nthose elements. If a server sends images, it should use the standard procedure for\nincorporating external data as described in Chapter 2. Servers are not required to support\nHTML or to send images, even if the client asks.
\n\n
A user or an FI can originate a message. E-mail messages are subject to\ndata synchronization so that a server can send a response again if it is lost or if it is\nused by multiple clients.
\n\n
Because e-mail messages cannot be replied to immediately, the response\nshould just echo back the original message (so that data synchronization will get this\noriginal e-mail message to other clients). When the FI is ready to reply, it should\ngenerate an unsolicited response (<TRNUID>0) and the client will pick this up during\nsynchronization.
Several services with Open Financial Exchange define e-mail requests and\nresponses that contain additional information specific to that service. To simplify\nimplementation for both clients and servers, this section defines a <MAIL> aggregate\nthat Open Financial Exchange uses in all e-mail requests and responses. For regular\ne-mail, the only additional information is an account from aggregate and whether to\ninclude images in the e-mail response or not. \n\n
Body of message, HTML-encoded or plain text depending on\n <USEHTML>, A-10000
\n
\n
\n
</MSGBODY>
\n
End of message
\n
\n
\n
<INCIMAGES>
\n
Include images in response, Boolean
\n
\n
\n
<USEHTML>
\n
Y if client wants an HTML response, N if client wants plain\n text, Boolean
\n
\n
\n
</MAIL>
\n
\n
\n
\n\n
If using HTML for the message body, clients and servers are REQUIRED to wrap the\ndesired HTML in an SGML marked section to protect the HTML markup: <![ CDATA [ ... html\n... ]]>. See the example.\n\n
E-mail is subject to synchronization. The transaction tag is\n<MAILTRNRQ> / <MAILTRNRS> and the synchronization tag is <MAILSYNCRQ> /\n<MAILSYNCRS>.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<MAILRQ>
\n
E-mail-message-request aggregate
\n
\n
\n
<MAIL>
\n
Core e-mail aggregate
\n
\n
\n
</MAIL>
\n
\n
\n
\n
</MAILRQ>
\n
\n
\n
\n\n
In a response, the <TRNUID> is zero if this is an unsolicited message. Otherwise,\nit should contain the <TRNUID> of the user's original message. It is RECOMMENDED\nthat servers include the <MESSAGE> of the user's message as part of the reply\n<MESSAGE>. The <MESSAGE> contents can include carriage returns to identify\ndesired line breaks.
Open Financial Exchange uses data synchronization to collect responses\nthat could have been lost due to communication problems, or that the servers previously\nsent to a different client or data file. All messages sent to the signed-on user ID are\ncovered by a single <TOKEN>. Note that this synchronization action expects only the\nbasic <MAILRS> responses. Specialized e-mail is received by means of their own\nsynchronization requests.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<MAILSYNCRQ>
\n
E-mail-synchronization-request aggregate
\n
\n
\n
<TOKEN>
\n
Client history marker
\n
\n
\n
<INCIMAGES>
\n
Include images in response, Boolean
\n
\n
\n
<USEHTML>
\n
Y if client wants an HTML response, N if client wants plain\n text, Boolean
In this example, a consumer requests information from customer service\nabout the checking statement just downloaded. This example omits the <OFX> top level\nand the signon <SONRQ>. This example uses HTML for the message body, and so it must\nprotect the HTML content in an SGML CDATA marked section.The request:
\n\n
<MAILTRNRQ>\n\t<TRNUID>54321\n\t<MAILRQ>\n\t\t<MAIL>\n\t\t\t<USERID>123456789\n\t\t\t<FROM>James Hackleman\n\t\t\t<TO>Noelani Federal Savings\n\t\t\t<SUBJECT>What do I need to earn interest?\n\t\t\t<DTCREATED>19960305\n\t\t\t<MSGBODY><![ CDATA [<HTML><BODY>I didn't earn any interest this month. Can you please tell me what I need to do to earn interest on this account?</BODY></HTML>\n]]></MSGBODY>\n\t\t\t<INCIMAGES>N\n\t\t\t<USEHTML>Y\n\t\t</MAIL>\n\t</MAILRQ>\n</MAILTRNRQ>The response from the FI:\n<MAILTRNRS>\n\t<TRNUID>54321\n\t<STATUS>\n\t\t<CODE>0\n\t\t<SEVERITY>INFO\n\t\t</STATUS>\n\t<MAILRS>\n\t\t<MAIL>\n\t\t\t<USERID>123456789\n\t\t\t<DTCREATED>19960307\n\t\t\t<FROM>Noelani Federal Savings\n\t\t\t<TO>James Hackleman\n\t\t\t<SUBJECT>Re: What do I need to earn interest?\n\t\t\t<MSGBODY>><![ CDATA [<HTML><BODY>You need to maintain $1000 in this account to earn interest. Because your balance was only $750 this month, no interest was earned. You could also switch to our new Checking Extra plan that always pays interest. Call us or check our web page http://www.fi.com/check-plans.html for more information.\nSincerely,\nCustomer Service Department\n\nOriginal message:\nI didn't earn any interest this month. Can you please tell me what I need to do to earn interest on this account?</BODY></HTML>\n]]></MSGBODY>\n\t\t\t<INCIMAGES>N\n\t\t\t<USEHTML>Y\n\t\t</MAIL>\n\t</MAILRS>\n</MAILTRNRS>\n
\n\n\n
Example of Synchronization Involving E-Mail
\n\n\n
In the following example the client did not receive the reply to the\nmessage sent in the previous example, so its <TOKEN> is one less than the server's.\nThe server replies by giving the current <TOKEN> and the missed response.
\n\n
<MAILSYNCRQ>\n\t<TOKEN>101\n</MAILSYNCRQ>\n\n<MAILSYNCRS>\n\t<TOKEN>102\n\t<MAILTRNRS>\n\t\t<TRNUID>54321\n\t\t<STATUS>\n\t\t\t<CODE>0\n\t\t\t<SEVERITY>INFO\n\t\t</STATUS>\n\t\t<MAILRS>\n\t\t\t... contents of e-mail message response as shown in previous example\n\t\t</MAILRS>\n\t</MAILTRNRS>\n</MAILSYNCRS>\n
Some responses contain values that are URLs, intended to be separately\nfetched by clients if desired. Clients can use their own HTTP libraries to perform this\nfetch outside of the Open Financial Exchange specification. However, to insulate clients\nagainst changes in transport technology, and to allow for fetches that require the\nprotection of an authenticated signon by a specific user, Open Financial Exchange defines\na transaction roughly equivalent to an HTTP Get. Any MIME type can be retrieved, including\nimages as well as HTML pages.\n\n
The following table lists the components of a request:
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<GETMIMERQ>
\n
Get-MIME-request aggregate
\n
\n
\n
<URL>
\n
URL, URL
\n
\n
\n
</GETMIMERQ>
\n
\n
\n
\n\n
The response simply echoes back the URL. The actual response, whether HTML, an image,\nor some other type, is always sent as a separate part of the file using multi-part MIME.
<GETMIMETRNRQ>\n\t<TRNUID>54321\n\t<GETMIMERQ>\n\t\t<URL>http://www.fi.com/apage.html\n\t</GETMIMERQ>\n</GETMIMETRNRQ>A response - the full\nfile is shown here to illustrate the use of multi-part MIME:\nHTTP 1.0 200 OK\nContent-Type: multipart/x-mixed-replace; boundary =--boundary-\n\n--boundary--\nContent-Type: application/x-ofx\nContent-Length: 8732\n\nOFXHEADER:100\nDATA:OFXSGML\nVERSION:100\nENCRYPTION:1\nENCODING:USASCII\n\n<OFX>\n\t\t<!-- signon not shown\n\t\tmessage set wrappers not shown -->\n<GETMIMETRNRS>\n\t<TRNUID>54321\n\t<STATUS>\n\t\t<CODE>0\n\t\t<SEVERITY>INFO\n\t</STATUS>\n\t<GETMIMERS>\n\t\t<URL>http://www.fi.com/apage.html\n\t</GETMIMERS>\n</GETMIMETRNRS>\n</OFX>\n\n--boundary--\nContent-Type: text/html\n<HTML>\n\t<!-- standard HTML page -->\n</HTML>\n\n--boundary--\n\n\n
If either or both of the messages in the e-mail message set are\nsupported, the following aggregate must be included in the profile <MSGSETLIST>\nresponse.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<EMAILMSGSET>
\n
E-mail-message-set-profile-information aggregate
\n
\n
\n
<EMAILMSGSETV1>
\n
Opening tag for V1 of the message set profile information\n
\n
\n
\n
<MSGSETCORE>
\n
Common message set information, defined in the profile\n chapter
Open Financial Exchange enables users to automate transactions that\noccur on a regular basis. Recurring transactions are useful when a customer has payments\nor transfers, for example, that repeat at regular intervals. The customer can create a\n\"model\" at the server for automatic generation of these instructions. The model\nin turn creates payments or transfers until it is canceled or expires. After the user\ncreates a recurring model at the server, the server can relieve the user from the burden\nof creating these transactions; it generates the transactions on its own, based on the\noperating parameters of the model.\n\n
The client must provide the following information to create a model:\n\n
\n
Type of transaction generated by the model (payment or transfer)
\n
Frequency of recurring transaction
\n
Total number of recurring transactions to generate
\n
Service-specific information, such as transfer date, payment amount,\n payee address
\n
\n\n
The model creates each transaction some time before its due date,\nusually thirty days. This allows the user to retrieve the transactions in advance of\nposting. This also gives the user the opportunity to modify or cancel individual\ntransactions without changing the recurring model itself.
\n\n
When a model is created, it can generate several transactions\nimmediately. The model does not automatically return responses for the newly created\ntransactions. It only returns a response to the request that was made to create the model.\nFor this reason, clients should send a synchronization request along with the request to\ncreate a model. This allows the server to return the newly created transaction responses,\nas well as the response to the request to set up a new model.\n\n
The Recurring Instructions aggregate is used to specify the schedule for\na repeating instruction. It is passed to the server when a recurring transfer or payment\nmodel is first created.
\n\n
\n
\n
Tag
\n
Description
\n
\n
\n
<RECURRINST>
\n
Recurring-Instructions aggregate
\n
\n
\n
<FREQ>
\n
Frequency, see section 10.2.1
\n
\n
\n
<NINSTS>
\n
Number of instructions
If this tag\n is absent, the schedule is open-ended, N-3
The following example illustrates the creation of a repeating payment.\nThe payment repeats on a monthly basis for 12 months. All payments are for $395.The\nrequest:
\n\n
.\n.\n.\n<RECPMTRQ>\n\t<RECURRINST>\n\t\t<FREQ>MONTHLY\n\t\t<NINSTS>12\n\t</RECURRINST>\n\t<PMTINFO>\n\t\t<BANKACCTFROM>\n\t\t\t<BANKID>555432180\n\t\t\t<ACCTID>763984\n\t\t\t<ACCTTYPE>CHECKING\n\t\t</BANKACCTFROM>\n\t\t<TRNAMT>395.00\n\t\t<PAYEEID>77810\n\t\t<PAYACCT>444-78-97572\n\t\t<DTDUE>19971115\n\t\t<MEMO>Auto loan payment\n\t</PMTINFO>\n</RECPMTRQ>\n.\n.\n.\n\nThe response includes the <RECSRVRTID>\nthat the client can use\n\nto cancel or modify the model:\n.\n.\n.\n<RECPMTRS>\n\t<RECSRVRTID>387687138\n\t<RECURRINST>\n\t\t<FREQ>MONTHLY\n\t\t<NINSTS>12\n\t</RECURRINST>\n\t<PMTINFO>\n\t\t<BANKACCTFROM>\n\t\t\t<BANKID>555432180\n\t\t\t<ACCTID>763984\n\t\t\t<ACCTTYPE>CHECKING\n\t\t</BANKACCTFROM>\n\t\t<TRNAMT>395.00\n\t\t<PAYEEID>77810\n\t\t<PAYACCT>444-78-97572\n\t\t<DTDUE>19971115\n\t\t<MEMO>Auto loan payment\n\t</PMTINFO>\n</RECPMTRS>\n.\n.\n.\n
Once created, a recurring model independently generates instructions.\nSince the client has not directly generated these transactions, the client has no record\nof their creation. To enable users to modify and/or cancel pending instructions, the\nclient must use data synchronization in order to retrieve these transactions.
\n\n
The client has two purposes for synchronizing state with the server with\nrespect to recurring models:\n\n
\n
Retrieve any added, modified, or canceled recurring models
\n
Retrieve any added, modified, or canceled transactions generated by any\n models
\n
\n\n
The client must be able to synchronize with the state of any models at\nthe server, as well as the state of any transactions generated by the server.\n\n
Once created and retrieved by the customer, recurring payments and\ntransfers are almost identical to customer-created payments or transfers. As with ordinary\npayments or transfers, you can cancel or modify transactions individually. However,\nbecause servers generate these transfers, they are different in the following respects:\n\n
\n
Recurring transactions must be retrieved as part of a synchronization\n request.
\n
Recurring transactions are related to a model. A server can modify or\n cancel transactions if the model is modified or canceled.
A recurring model can be modified or canceled. When a model is modified,\nall transactions that it generates in the future will change as well. The client can\nindicate whether transactions that have been generated, but have not been sent, should be\nmodified as well. The actual elements within a transaction that can be modified differ by\nservice. See the recurring sections within the Banking and Payments chapters for details.\n
\n\n
A user can cancel a model immediately or at a future date. If a user\ncancels the model immediately, the client cancels any transactions that it has not yet\nsent. If the client schedules the cancel for a future date, the client will not cancel\npending transactions.\n\n
Canceling a recurring payment model requires the client to pass the\n<RECSRVRTID> of the model. The client requests that pending payments also be\ncanceled. The server cancels the model immediately and notifies the client that both the\nmodel and any scheduled payments were canceled.The request:
The server also cancels any payments that have been generated but not\nexecuted. In the example shown above, the client would not learn of this immediately. To\nreceive notification that the model and all generated payments were canceled, the client\nwould need to include a synchronization request in the file. The following example\nillustrates this alternate approach.The request file now includes a synchronization\nrequest:
\n\n
.\n.\n.\n\t<RECPMTCANCRQ>\n\t\t<RECSRVRTID>387687138\n\t\t<CANPENDING>Y\n\t</RECPMTCANCRQ>\n\t<PMTSYNCRQ>\n\t\t<TOKEN>12345\n\t\t<BANKACCTFROM>\n\t\t\t<BANKID>123432123\n\t\t\t<ACCTID>516273\n\t\t\t<ACCTTYPE>CHECKING\n\t\t</BANKACCTFROM>\n\t</PMTSYNCRQ>\n.\n.\n.\nThe response file now contains two responses\n(assuming one payment was pending),\n\none for the canceled model and one for the canceled payment.\n.\n.\n.\n\t<RECPMTCANCRS>\n\t\t<RECSRVRTID>387687138\n\t\t<CANPENDING>Y\n\t</RECPMTCANCRS>\n\t<PMTSYNCRS>\n\t\t<TOKEN>3247989384\n\t\t<BANKACCTFROM>\n\t\t\t<BANKID>123432123\n\t\t\t<ACCTID>516273\n\t\t\t<ACCTTYPE>CHECKING\n\t\t</BANKACCTFROM>\n\t\t<PMTTRNRS>\n\t\t\t<TRNUID>10103\n\t\t\t<STATUS>\n\t\t\t\t<CODE>0\n\t\t\t\t<SEVERITY>INFO\n\t\t\t</STATUS>\n\t\t\t<PMTCANCRS>\n\t\t\t\t<SRVRTID>1030155\n\t\t\t</PMTCANCRS>\n\t\t</PMTTRNRS>\n\t</PMTSYNCRS>\n.\n.\n.\n\n\n
![\"Quivr-logo\"](\"https://raw.githubusercontent.com/QuivrHQ/MegaParse/main/logo.png\"){kind=logo}
\n![\"stadyn_image1.gif](\"./sample_complexe_files/stadyn_image1.gif\")
![\"stadyn_image2.gif](\"./sample_complexe_files/stadyn_image2.gif\")
![\"stadyn_image3.gif](\"./sample_complexe_files/stadyn_image3.gif\")
![\"stadyn_image4.gif](\"./sample_complexe_files/stadyn_image4.gif\")
![\"stadyn_image5.gif](\"./sample_complexe_files/stadyn_image5.gif\")
![\"stadyn_image6.gif](\"./sample_complexe_files/stadyn_image6.gif\")
![\"stadyn_image7.gif](\"./sample_complexe_files/stadyn_image7.gif\")
![\"stadyn_image8.gif](\"./sample_complexe_files/stadyn_image8.gif\")
![\"stadyn_image9.gif](\"./sample_complexe_files/stadyn_image9.gif\")
\nstadyn_image9![\"stadyn_image10.gif](\"./sample_complexe_files/stadyn_image10.gif\")