README.rst 3.77 KB
Newer Older
M. Huang's avatar
M. Huang committed
1
FDI helps data producers and processors to build connections into isolated heterogeneous datasets. to assemble, organize, and integrate data into self-describing, modular, hierarchical, persistent, referenceable ``Products``, whose component datasets keep their own characteristicss and are easily accessible.
2 3

FDI provides scripting-friendly  APIs  and 
M. Huang's avatar
M. Huang committed
4
tools to define custom Products and generating Python class files. An integrated Product takes care of inter-platform compatibility, string representation, serialisation to simple exchange format, persistence to disk or server, and carrying references of other Products, enabling context-building and lazy-loading.
5 6 7

FDI's base data model is defined in package ``dataset``. Persistent data
access, referencing, and Universal Resource Names are defined in package
M. Huang's avatar
M. Huang committed
8
``pal``. A reference RESTful API server designed to communicate with a data
9 10 11 12 13 14 15 16 17 18 19
processing server/docker using the data model, and a reference data store (``pool``) server is in package ``pns``.

Install/Uninstall

For Users
=========

To install (It is a good idea to add ``--user`` at the end or to use a virtualenv to avoid disturbing Python setup.)

.. code-block:: shell

M. Huang's avatar
v1.0  
M. Huang committed
20
   python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/v1.0/fdi-v1.0rc.tar.gz
21

22 23 24 25
If you want to install the ``develop`` branch:

.. code-block:: shell

M. Huang's avatar
v1.0  
M. Huang committed
26
   python3 -m pip install git+http://mercury.bao.ac.cn:9006/mh/fdi.git@develop
27
   
28 29 30 31
To uninstall:

.. code-block:: shell

M. Huang's avatar
v1.0  
M. Huang committed
32
           python3 -m pip uninstall fdi
33 34 35 36 37 38 39 40 41 42 43


For Developers and Admins
=========================

To install (It is a good idea to add ``--user`` at the end or to use a virtualenv to avoid disturbing Python setup.)

.. code-block:: shell

           FDIINSTDIR=/tmp   # change this to your installation dir
           cd $FDIINSTDIR
44
           git clone ssh://git@mercury.bao.ac.cn:9005/mh/fdi.git@develop
45
           cd fdi
M. Huang's avatar
v1.0  
M. Huang committed
46
           python3 -m pip install -e .[DEV]
47

48
If you want to install the ``master`` branch, remove the ``@develop`` part above..   
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93
	   
To test your installation:

.. code-block:: shell

           make test

.. tip::

   To pass command-line arguments to ``pytest`` do, for example,
   
   .. code-block:: shell
		   
		make test T='-k Bas'

   to test ``BaseProduct`` in sub-package ``dataset``.

To generate ``baseproduct.py`` and ``product.py`` from YAML schema files in
``fdi/dataset/resources``:

.. code-block:: shell

           make py

Modify/Generate Documents
-------------------------

If you plan to compile documents in the ``doc`` directory, generate diagrams, API files, or HTML pages, run (in that order, respectively):

.. code-block:: shell

           make doc_plots
           make doc_api
           make doc_html

.. note:: Read-the-docs makes web pages from sources in ``doc/sphinx`` in the repository. Locally generated HTML pages are not on RTD or in the repository. The API files and plots, however need to go to the repo.
	   
Run Servers
-----------

If you plan to run the ``pns`` and/or the http pool server locally,
install the dependencies:

.. code-block:: shell

M. Huang's avatar
v1.0  
M. Huang committed
94
           python3 -m pip install -e .[SERV]
95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
	   make installpns

To test your ``pns`` servers installation, in one window, run:

.. code-block:: shell

           make runserver

in another window run:

.. code-block:: shell

           make testpns

To test your ``httppool`` servers installation, in one window, run:

.. code-block:: shell

           make runpoolserver

in another window run:

.. code-block:: shell

           make testhttppool

For more examples see ``tests/test_*.py``

Read more on package introduction, description, quick start, and API
documents on `readthedocs.io <https://fdi.readthedocs.io/en/latest/>`__.