Commit a5af87ee authored by M. Huang's avatar M. Huang
Browse files

context inherit BaseProduct to allow application projects to put parameter in...

context inherit BaseProduct to allow application projects to put parameter in Products in other classes. YAML schema to v1.5 for dataset. v1.5.4
parent 255f240e
docs/sphinx/_static/product.png

35.3 KB | W: | H:

docs/sphinx/_static/product.png

50.8 KB | W: | H:

docs/sphinx/_static/product.png
docs/sphinx/_static/product.png
docs/sphinx/_static/product.png
docs/sphinx/_static/product.png
  • 2-up
  • Swipe
  • Onion skin
......@@ -20,11 +20,14 @@ A product has
* zero or more datasets: defining well described data entities (say images, tables, spectra etc...).
* accompanying meta data -- required information such as
* who created this product,
* what does the data reflect (say measurement from an instrument)
* the classification of this product,
* the creator this product,
* when was the product created?
* what does the data reflect? (its intended use scope)
* and so on;
* possible additional meta data specific to that particular product type.
* possible additional meta data specific to that particular product type.
* history of this product: how was this data created.
* References of relevant products that form a context of this product
History
......@@ -41,10 +44,14 @@ In order to transfer data across the network between heterogeneous nodes data ne
JSON format is used considering to transfer serialized data for its wide adoption, availability of tools, ease to use with Python, and simplicity.
Defining Products
-----------------
Product Definition Methodology
------------------------------
A number of built-in Parameters can be specified in ``fdi/dataset/resourcese`` in YAML format. A helper utility ``yaml2python`` can be run using ``make py`` to generate test-ready Python code of product class module containing the built-ins.
Data Products almost always are classified in heirachical orders, reflecting the underlying relation of the data model. Many Products are found to have inheritance relations when comparing their metadata and datasets. Therefore an object-oriented approach is chosen to anlize and define the structure, function, and interface of Products here.
First specify built-in Parameters in in YAML format, which is suitable for reading by both humans and machines. A helper utility ``yaml2python`` to generate test-ready Python code of product class module containing the built-ins.
The YAML schema allows a child Product to inherit metadata definition from one or multiple parent Products. Overriding is also allowed.
BaseProduct
"""""""""""
......@@ -70,6 +77,10 @@ From the creation process requires every product to carry the following metadata
:FORMATV: -- Version of this document with Schema information, e.g. 1.4.1.2
:creator, rootCause, creationDate: -- Who, why, when, where
The parameters are tabulated below.
.. include:: ../../../fdi/dataset/resources/BaseProduct.txt
:code: text
Examples (from :doc:`quickstart` page):
......
__version_info__ = (1, 5, 3)
__version_info__ = (1, 5, 4)
__version__ = '.'.join(map(str, __version_info__))
......@@ -30,7 +30,7 @@ _Model_Spec = {
None,
],
'level': 'ALL',
'schema': '1.4',
'schema': '1.5',
'metadata': {
'description': {
'id_zh_cn': '描述',
......@@ -92,7 +92,7 @@ _Model_Spec = {
'data_type': 'string',
'description': 'Version of product schema and revision',
'description_zh_cn': '产品格式版本',
'default': '1.4.0.8',
'default': '1.5.0.9',
'valid': '',
'typecode': 'B',
},
......@@ -131,9 +131,9 @@ class BaseProduct( AbstractComposite, Copyable, Serializable, EventSender):
p.meta['creator']=Parameter('bar')
assert p.meta['creator']==Parameter('bar')
BaseProduct class (level ALL) schema 1.4 inheriting [None].
BaseProduct class (level ALL) schema 1.5 inheriting [None].
Automatically generated from fdi/dataset/resources/BaseProduct.yml on 2021-05-11 07:28:00.012646.
Automatically generated from fdi/dataset/resources/BaseProduct.yml on 2021-05-21 15:09:03.057604.
Description:
FDI base class
......@@ -147,7 +147,7 @@ FDI base class
creationDate = FineTime(0),
rootCause = 'UNKNOWN',
version = '0.8',
FORMATV = '1.4.0.8',
FORMATV = '1.5.0.9',
zInfo=None,
**kwds):
......
......@@ -25,7 +25,7 @@ _Model_Spec = {
'BaseProduct',
],
'level': 'ALL',
'schema': '1.4',
'schema': '1.5',
'metadata': {
'description': {
'id_zh_cn': '描述',
......@@ -88,7 +88,7 @@ _Model_Spec = {
'data_type': 'string',
'description': 'Version of product schema and revision',
'description_zh_cn': '产品格式版本',
'default': '1.4.0.8',
'default': '1.5.0.9',
'valid': '',
'typecode': 'B',
},
......@@ -154,9 +154,9 @@ _Model_Spec = {
ProductInfo = ReadOnlyDict(_Model_Spec)
class Product(BaseProduct):
""" Product class (level ALL) schema 1.4 inheriting ['BaseProduct'].
""" Product class (level ALL) schema 1.5 inheriting ['BaseProduct'].
Automatically generated from fdi/dataset/resources/Product.yml on 2021-05-11 07:28:00.149739.
Automatically generated from fdi/dataset/resources/Product.yml on 2021-05-21 15:09:03.125711.
Description:
Project level product
......@@ -172,7 +172,7 @@ Project level product
creationDate = FineTime(0),
rootCause = 'UNKNOWN',
version = '0.8',
FORMATV = '1.4.0.8',
FORMATV = '1.5.0.9',
startDate = FineTime(0),
endDate = FineTime(0),
instrument = 'UNKNOWN',
......
......@@ -3,7 +3,7 @@ description: FDI base class
parents:
-
level: ALL
schema: '1.4'
schema: '1.5'
metadata:
description:
id_zh_cn: 描述
......@@ -59,6 +59,8 @@ metadata:
data_type: string
description: Version of product schema and revision
description_zh_cn: 产品格式版本
default: '1.4.0.8'
default: '1.5.0.9'
valid: ''
typecode: B
datasets:
......@@ -3,7 +3,7 @@ description: Project level product
parents:
- BaseProduct
level: ALL
schema: '1.4'
schema: '1.5'
metadata:
type:
id_zh_cn: 产品类型
......@@ -76,6 +76,8 @@ metadata:
data_type: string
description: Version of product schema and revision
description_zh_cn: 产品格式版本
default: '1.4.0.8'
default: '1.5.0.9'
valid: ''
typecode: B
datasets:
......@@ -635,9 +635,9 @@ if __name__ == '__main__':
prod = glb[prodname]()
fg = {'name': 15, 'value': 18, 'unit': 7, 'type': 8,
'valid': 26, 'default': 18, 'code': 4, 'description': 25}
sp = prod.meta.toString(tablefmt='fancy_grid', widths=fg)
sp = prod.meta.toString(tablefmt='rst', widths=fg)
mout = pathjoin(ypath, prodname + '.txt')
mout = pathjoin(ypath, prodname + '.rst')
with open(mout, 'w', encoding='utf-8') as f:
f.write(sp)
print('Done dumping ' + mout + '\n' + '*'*40)
......
# -*- coding: utf-8 -*-
from ..dataset.serializable import Serializable
from ..dataset.product import Product
from ..dataset.baseproduct import BaseProduct
from ..dataset.odict import ODict
import logging
# create logger
......@@ -241,7 +241,7 @@ http://herschel.esac.esa.int/hcss-doc-15.0/load/hcss_drm/api/herschel/ia/pal/Con
return True
class Context(AbstractContext, Product):
class Context(AbstractContext, BaseProduct):
""" See docstring of AbstractContext.
"""
......@@ -359,7 +359,7 @@ class MapContext(Context):
2. the put() method of the map view may throw a ValueError if either of the arguments to the put() method are null.
BaseProduct--Product
BaseProduct--Product
\\
AbstractContext------Contex---------MapContext
......
......@@ -50,7 +50,7 @@ setup(
'waitress>=2.0.0'
],
'PUB': [
'aiohttp>=3.6.2',
# 'aiohttp>=3.6.2',
'sphinx_rtd_theme>=0.4.3',
'sphinx-copybutton>=0.3.0',
'twine>=3.3.0'
......
......@@ -19,6 +19,7 @@ from fdi.pal.context import Context
from fdi.pal.query import AbstractQuery, MetaQuery
from fdi.dataset.deserialize import deserialize
from fdi.dataset.product import Product
from fdi.dataset.baseproduct import BaseProduct
from fdi.dataset.eq import deepcmp
from fdi.dataset.classes import Classes
from fdi.dataset.metadata import MetaData, Parameter
......@@ -714,13 +715,13 @@ def doquery(poolpath, newpoolpath):
for i in range(n):
a0, a1, a2 = 'desc %d' % i, 'fatman %d' % (i*4), 5000+i
if i < 3:
x = TP(description=a0, instrument=a1)
x = TP(description=a0, creator=a1)
x.meta['extra'] = Parameter(value=a2)
elif i < 5:
x = Context(description=a0, instrument=a1)
x = Context(description=a0, creator=a1)
x.meta['extra'] = Parameter(value=a2)
else:
x = MapContext(description=a0, instrument=a1)
x = MapContext(description=a0, creator=a1)
x.meta['extra'] = Parameter(value=a2)
x.meta['time'] = Parameter(value=FineTime1(a2))
if i < 4:
......@@ -742,19 +743,19 @@ def doquery(poolpath, newpoolpath):
p = r.product
assert type(p) == type(c['p'])
assert p.description == c['a0']
assert p.instrument == c['a1']
assert p.creator == c['a1']
assert p.meta['extra'].value == c['a2']
chk(res[0], rec1[m])
# query with a parent class and a specific parameter
m = 3
q = MetaQuery(Product, 'm["instrument"].value == "%s"' % rec1[m]['a1'])
q = MetaQuery(BaseProduct, 'm["creator"].value == "%s"' % rec1[m]['a1'])
res = pstore.select(q)
assert len(res) == 1, str(res)
chk(res[0], rec1[m])
# query with a parent class and a specific parameter
q = MetaQuery(Product, 'm["extra"].value < 5002')
q = MetaQuery(BaseProduct, 'm["extra"].value < 5002')
res = pstore.select(q)
# [0,1]
assert len(res) == 2, str(res)
......@@ -764,7 +765,7 @@ def doquery(poolpath, newpoolpath):
# simpler syntax for comparing value only but a bit slower.
# the parameter with simpler syntax must be on the left hand side of a comparison operator.
# '5000 < m["extra"]' does not work. But '5000 < m["extra"].value' works.
q = MetaQuery(Product, 'm["extra"] > 5000 and m["extra"] <= 5002')
q = MetaQuery(BaseProduct, 'm["extra"] > 5000 and m["extra"] <= 5002')
res = pstore.select(q)
# [1,2]
assert len(res) == 2, str(res)
......@@ -772,7 +773,7 @@ def doquery(poolpath, newpoolpath):
chk(res[1], rec1[2])
# two classes
q = MetaQuery(Product, 'm["extra"] > 5000 and m["extra"] < 5004')
q = MetaQuery(BaseProduct, 'm["extra"] > 5000 and m["extra"] < 5004')
res = pstore.select(q)
# [1,2,3]
assert len(res) == 3, str(res)
......@@ -781,20 +782,20 @@ def doquery(poolpath, newpoolpath):
chk(res[2], rec1[3])
# this is not in this store
q = MetaQuery(Product, 'm["extra"] == 5004')
q = MetaQuery(BaseProduct, 'm["extra"] == 5004')
res = pstore.select(q)
# []
assert len(res) == 0, str(res)
# it is in the other store
q = MetaQuery(Product, 'm["extra"] == 5004')
q = MetaQuery(BaseProduct, 'm["extra"] == 5004')
res = pstore2.select(q)
# [4]
assert len(res) == 1, str(res)
chk(res[0], rec1[4])
# all in the other store
q = MetaQuery(Product, '1')
q = MetaQuery(BaseProduct, '1')
res = pstore2.select(q)
# [4,5,6]
assert len(res) == 3, str(res)
......@@ -830,15 +831,15 @@ def doquery(poolpath, newpoolpath):
chk(res[0], rec1[5])
# all 'extra' < 5002, all in 1st pool
q = MetaQuery(Product, 'm["extra"] < 5002')
q = MetaQuery(BaseProduct, 'm["extra"] < 5002')
res = pstore.select(q)
# [0,1 ]
assert len(res) == 2, str(res)
chk(res[0], rec1[0])
chk(res[1], rec1[1])
# instrument = 'fatman 12|16', two pools
q = MetaQuery(Product, '"n 1" in m["instrument"].value')
# creator = 'fatman 12|16', two pools
q = MetaQuery(BaseProduct, '"n 1" in m["creator"].value')
res = pstore.select(q)
# [3,4]
assert len(res) == 2, str(res)
......@@ -849,9 +850,9 @@ def doquery(poolpath, newpoolpath):
# same as above but query is a function
def t(m):
import re
return re.match('.*n.1.*', m['instrument'].value)
return re.match('.*n.1.*', m['creator'].value)
q = MetaQuery(Product, t)
q = MetaQuery(BaseProduct, t)
res = pstore.select(q)
# [3,4]
assert len(res) == 2, str(res)
......@@ -859,7 +860,7 @@ def doquery(poolpath, newpoolpath):
chk(res[1], rec1[4])
# same as above but query is on the product. this is slow.
q = AbstractQuery(Product, 'p', '"n 1" in p.instrument')
q = AbstractQuery(BaseProduct, 'p', '"n 1" in p.creator')
res = pstore.select(q)
# [3,4]
assert len(res) == 2, str(res)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment