gudusoft.gsqlparser.dlineage.util

Class DataflowUtility

Object

gudusoft.gsqlparser.dlineage.util.DataflowUtility

public class DataflowUtility
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataflowUtility.GroupNames How a merge names the table it makes of a group of tables: which member it takes its properties from (DataflowUtility.GroupNames.displayed), which table's names it publishes (DataflowUtility.GroupNames.names), and whether the group spans legacy folded names, which only identities join (DataflowUtility.GroupNames.acrossLegacyNames).
static class	DataflowUtility.SpilledIdentities What a spilled model's physical tables and views carried that XML does not, by table id: the identities their models minted, which tables publish a reader's vouched SQL spelling (table.isPublishedNameSqlSpelling()), the minted names a merged table carries (table.getMintedNames()), which merged tables span legacy folded names (table.isAcrossLegacyNames()), and the exact names a merged table's carried tables published (table.getCarriedPublishedNames()).

Constructor Summary

Constructors

Constructor and Description
DataflowUtility()

Method Summary

Modifier and Type	Method and Description
static Object	calledRoutineKey(String callName) The routineKey(java.lang.String) of the routine a call names: the call's name qualified by the default database and schema as written (DlineageUtil.getTableFullNameAsWritten(java.lang.String)), the way the merge has always qualified it.
static DataflowUtility.SpilledIdentities	carriedIdentities(dataflow df) Captures the DataflowUtility.SpilledIdentities of a dataflow about to be serialized.
static void	carryGroupIdentity(table merged, List<table> group, DataflowUtility.GroupNames chosen) Gives a merged table the identity of the tables it replaces (the one a model minted), the marker of the table its names come from (whether those names are a reader's vouched SQL spelling), when those names are not ones a model minted the first minted names the group offers, and whether the group spans legacy folded names.
static dataflow	cloneDataflow(dataflow dataflow)
static List<Object>	columnsOfTable(Collection<List<column>> columnsByFoldedName) What the columns of one output table become when the elements that publish it are merged: in order, a function column as it is, or the columns that are one column.
static dataflow	convertTableLevelToFunctionCallDataflow(dataflow dataflow, boolean showBuiltIn, EDbVendor dbVendor)
static dataflow	convertToSchemaLevelDataflow(dataflow dataflow, EDbVendor dbVendor)
static dataflow	convertToSchemaLevelDataflow(dataflow dataflow, EDbVendor dbVendor, boolean isSimple)
static dataflow	convertToTableLevelDataflow(dataflow dataflow)
static Map<String,table>	getDataflowDbObjMap(dataflow dataflow)
static Map<String,table>	getDataflowDbObjNameMap(dataflow dataflow)
static QualifiedCanonKey	groupIdentity(List<table> merged) The identity a model minted for any of the tables merged into one; they share one merge key.
static DataflowUtility.GroupNames	groupNames(List<table> group) How a merge names the table it makes of group, in the group's order; see DataflowUtility.GroupNames.
static CanonKey	identityOf(List<column> group) The identity a group of columns stands for: the one its named columns carry, or null for a star or system group and for named columns that carry none.
static dataflow	iterativeMergeDataflowsFromFiles(List<File> tempFiles, EDbVendor vendor) 从一组临时 XML 文件迭代加载并合并 dataflow，节约内存。
static dataflow	iterativeMergeDataflowsFromFilesByStartId(List<File> tempFiles, long startId) 从一组临时 XML 文件迭代加载并合并 dataflow，调用方需保证文件中的 dataflow ID 空间互不冲突， 且已设置好 ModelBindingManager。
static dataflow	iterativeMergeDataflowsFromFilesByStartId(List<File> tempFiles, long startId, BiConsumer<File,dataflow> afterLoad) The same merge, calling afterLoad with each file's dataflow as soon as it is read, before any id is remapped: a spilled model's same-run state that XML does not carry (its tables' carried identities, see carriedIdentities(gudusoft.gsqlparser.dlineage.dataflow.model.xml.dataflow)) is put back there.
static void	labelEndpoints(dataflow flow) labelEndpoints(gudusoft.gsqlparser.dlineage.dataflow.model.xml.relationship, java.util.Map<java.lang.String, java.lang.String>) for every relationship of flow, once its merge is done; then a relationship that now repeats another in every attribute but its id (exactKey(gudusoft.gsqlparser.dlineage.dataflow.model.xml.relationship): coordinates and the endpoints' ids included, so no evidence site and no edge is lost) is dropped, and the comment bindings that named it follow the one kept.
static void	labelEndpoints(relationship r, Map<String,String> publishedColumnNames) Labels each endpoint of r with the published name of the column its ids name (publishedColumnNames(gudusoft.gsqlparser.dlineage.dataflow.model.xml.dataflow)), so the label of an endpoint and the name of the column it points to never disagree: a merge that makes one column of several keeps one name, and the endpoints that named the others by their own spellings (a star link's folded display, the spelling of another reference) now carry it too.
static dataflow	mergeDataflows(Collection<dataflow> dataflows, EDbVendor vendor) 合并一组 startId 都为 0（即 ID 空间相互冲突）的 dataflow。
static dataflow	mergeDataflows(Collection<dataflow> dataflows, EDbVendor vendor, IdentifierProfile identifierProfile) Merge under an explicit identifier configuration — the configuration the analyses that produced dataflows ran under.
static Dataflow	mergeDataflowsAndCsv(List<Pair<EDbVendor,dataflow>> pairs, String csvMetadata)
static dataflow	mergeDataflowsByStartId(Collection<dataflow> dataflows, long startId) 内部合并实现：调用方需保证所有输入 dataflow 的 ID 空间不冲突，并且已提前设置好 ModelBindingManager。
static dataflow	mergeDataflowsWithDifferentStartId(Collection<dataflow> dataflows, EDbVendor vendor)
static dataflow	mergeDataflowsWithDifferentStartId(Collection<dataflow> dataflows, EDbVendor vendor, IdentifierProfile identifierProfile) As mergeDataflowsWithDifferentStartId(Collection, EDbVendor), under an explicit identifier configuration.
static dataflow	mergeFunctionCallDataflow(dataflow dataflow, EDbVendor dbVendor)
static Map<String,String>	nameCollidingIdentities(List<column> columns, EDbVendor vendor) Names that tell the columns of one table apart.
static Map<String,String>	publishedColumnNames(dataflow flow) The published name of every column of flow, by the column's parent and its own id: what a relationship endpoint that names the column by those ids is labelled (labelEndpoints(gudusoft.gsqlparser.dlineage.dataflow.model.xml.relationship, java.util.Map<java.lang.String, java.lang.String>)).
static dataflow	readDataflowFromCsvMetadata(String csvMetadata)
static dataflow	readDataflowFromCsvMetadata(String csvMetadata, EDbVendor vendor)
static void	renameEndpoints(List<relationship> relationships, Map<String,String> renamed)
static void	restoreCarriedIdentities(dataflow df, DataflowUtility.SpilledIdentities identities)
static Object	routineKey(String name) The identity a routine and a call of it meet under in a merge, from a name as written: keyed as a function name by the analysis's identifier rules (DlineageUtil.lexicalNameKey(gudusoft.gsqlparser.EDbVendor, java.lang.String, gudusoft.gsqlparser.sqlenv.ESQLDataObjectType)).
static String	sqlSpellingOf(List<column> group) The spelling as SQL one of the columns that are one column knows, or null.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataflowUtility

public DataflowUtility()

Method Detail

mergeFunctionCallDataflow

public static dataflow mergeFunctionCallDataflow(dataflow dataflow,
                                                 EDbVendor dbVendor)

convertTableLevelToFunctionCallDataflow

public static dataflow convertTableLevelToFunctionCallDataflow(dataflow dataflow,
                                                               boolean showBuiltIn,
                                                               EDbVendor dbVendor)

convertToTableLevelDataflow

public static dataflow convertToTableLevelDataflow(dataflow dataflow)

convertToSchemaLevelDataflow

public static dataflow convertToSchemaLevelDataflow(dataflow dataflow,
                                                    EDbVendor dbVendor)
                                             throws Exception

Throws:

Exception

convertToSchemaLevelDataflow

public static dataflow convertToSchemaLevelDataflow(dataflow dataflow,
                                                    EDbVendor dbVendor,
                                                    boolean isSimple)
                                             throws Exception

Throws:

Exception

cloneDataflow

public static dataflow cloneDataflow(dataflow dataflow)

getDataflowDbObjMap

public static Map<String,table> getDataflowDbObjMap(dataflow dataflow)

getDataflowDbObjNameMap

public static Map<String,table> getDataflowDbObjNameMap(dataflow dataflow)

mergeDataflowsWithDifferentStartId

public static dataflow mergeDataflowsWithDifferentStartId(Collection<dataflow> dataflows,
                                                          EDbVendor vendor)

mergeDataflowsWithDifferentStartId

public static dataflow mergeDataflowsWithDifferentStartId(Collection<dataflow> dataflows,
                                                          EDbVendor vendor,
                                                          IdentifierProfile identifierProfile)

As mergeDataflowsWithDifferentStartId(Collection, EDbVendor), under an explicit identifier configuration.

Parameters:

dataflows - results whose ID spaces already do not overlap

vendor - dialect of the results

identifierProfile - the configuration the results were produced under; null means the vendor default

Returns:

the merged result

mergeDataflowsByStartId

public static dataflow mergeDataflowsByStartId(Collection<dataflow> dataflows,
                                               long startId)

内部合并实现：调用方需保证所有输入 dataflow 的 ID 空间不冲突，并且已提前设置好 ModelBindingManager。

该方法代替之前位于 ParallelDataFlowAnalyzer 的同名实现，外部调用者应使用 mergeDataflowsWithDifferentStartId(Collection, EDbVendor) 或 mergeDataflows(Collection, EDbVendor)。

mergeDataflows

public static dataflow mergeDataflows(Collection<dataflow> dataflows,
                                      EDbVendor vendor)

合并一组 startId 都为 0（即 ID 空间相互冲突）的 dataflow。

标准合并 mergeDataflowsWithDifferentStartId(Collection, EDbVendor) 的前提是 每个 dataflow 在生成时已通过 optionCopy.setStartId(5000000L * i) 占用了 不重叠的 ID 段；当外部调用方拿到的 dataflow 都是用默认 startId=0 生成时，直接合并 会因为 ID 冲突而错乱。本方法会先按下标把每个 dataflow 的全部 ID 偏移 DATAFLOW_ID_RANGE * index，再委托给标准合并逻辑。

注意：此方法会原地修改入参 dataflow 的 ID（与标准合并行为一致）。

mergeDataflows

public static dataflow mergeDataflows(Collection<dataflow> dataflows,
                                      EDbVendor vendor,
                                      IdentifierProfile identifierProfile)

Merge under an explicit identifier configuration — the configuration the analyses that produced dataflows ran under.

The merge compares table and column identities, so it has to apply the same rules the results were produced with. The (Collection, EDbVendor) overload states the vendor DEFAULT; pass the profile here when the analyses were configured.

Parameters:

dataflows - results to merge, in source order

vendor - dialect of the results

identifierProfile - the configuration the results were produced under; null means the vendor default

Returns:

the merged result, or null for an empty input

Throws:

IllegalStateException - if an input carries provenance of a different configuration

iterativeMergeDataflowsFromFiles

public static dataflow iterativeMergeDataflowsFromFiles(List<File> tempFiles,
                                                        EDbVendor vendor)

从一组临时 XML 文件迭代加载并合并 dataflow，节约内存。

与 iterativeMergeDataflowsFromFilesByStartId(List, long) 不同的是，本方法假定文件中的 dataflow 是用默认 startId=0 生成的（ID 空间互相冲突），因此会按下标先做 ID 偏移再合并。

注意：本方法按入参 tempFiles 顺序迭代合并，不做任何排序。 coordinate 第三维 fileIdx 的偏移也是严格按入参顺序累积的， 调用方传入的顺序应与原始源文件的语义顺序保持一致。

iterativeMergeDataflowsFromFilesByStartId

public static dataflow iterativeMergeDataflowsFromFilesByStartId(List<File> tempFiles,
                                                                 long startId)

从一组临时 XML 文件迭代加载并合并 dataflow，调用方需保证文件中的 dataflow ID 空间互不冲突， 且已设置好 ModelBindingManager。

主要供 ParallelDataFlowAnalyzer 等内部生成方使用：每个临时文件中的 dataflow 已通过 optionCopy.setStartId(DATAFLOW_ID_RANGE * i) 占用了不重叠的 ID 段。

注意：本方法按入参 tempFiles 顺序迭代合并，不做任何排序。 coordinate 第三维 fileIdx 的偏移严格按入参顺序累积。

iterativeMergeDataflowsFromFilesByStartId

public static dataflow iterativeMergeDataflowsFromFilesByStartId(List<File> tempFiles,
                                                                 long startId,
                                                                 BiConsumer<File,dataflow> afterLoad)

The same merge, calling afterLoad with each file's dataflow as soon as it is read, before any id is remapped: a spilled model's same-run state that XML does not carry (its tables' carried identities, see carriedIdentities(gudusoft.gsqlparser.dlineage.dataflow.model.xml.dataflow)) is put back there.

columnsOfTable

public static List<Object> columnsOfTable(Collection<List<column>> columnsByFoldedName)

What the columns of one output table become when the elements that publish it are merged: in order, a function column as it is, or the columns that are one column. The columns come grouped under their folded names, in the order the names first appear. Under one folded name a star, a system column and a named column are never one column, and named columns are one column unless they carry different identities (identityGroups(java.util.List<gudusoft.gsqlparser.dlineage.dataflow.model.xml.column>)); columns of one identity and one qualifier are one column whatever their names.

Parameters:

columnsByFoldedName - the columns of the table under each folded name

Returns:

each element is a function column or a List<column> that is one column

nameCollidingIdentities

public static Map<String,String> nameCollidingIdentities(List<column> columns,
                                                         EDbVendor vendor)

Names that tell the columns of one table apart. Where columns of different identities have names that collide under the collision key of NormalNameCollisions (which ignores case), each of them that knows its spelling as SQL (column.getSqlSpelling()) is renamed as scanned collisions are published: the column whose identity the name denotes keeps it, another gets its stored or delimited name (PostgreSQL "Amount" beside AMOUNT, Oracle "Amount" beside AMOUNT).

Returns:

the new name of each renamed column, by column id; the caller renames the relationships naming them

sqlSpellingOf

public static String sqlSpellingOf(List<column> group)

The spelling as SQL one of the columns that are one column knows, or null.

routineKey

public static Object routineKey(String name)

The identity a routine and a call of it meet under in a merge, from a name as written: keyed as a function name by the analysis's identifier rules (DlineageUtil.lexicalNameKey(gudusoft.gsqlparser.EDbVendor, java.lang.String, gudusoft.gsqlparser.sqlenv.ESQLDataObjectType)). The analysis takes it when it publishes a routine or a call (procedure.getRoutineKey(), table.getRoutineKey()), before normalized output renames them; a merge reads a published name only where nothing carries it (a document read back from XML).

calledRoutineKey

public static Object calledRoutineKey(String callName)

The routineKey(java.lang.String) of the routine a call names: the call's name qualified by the default database and schema as written (DlineageUtil.getTableFullNameAsWritten(java.lang.String)), the way the merge has always qualified it.

publishedColumnNames

public static Map<String,String> publishedColumnNames(dataflow flow)

The published name of every column of flow, by the column's parent and its own id: what a relationship endpoint that names the column by those ids is labelled (labelEndpoints(gudusoft.gsqlparser.dlineage.dataflow.model.xml.relationship, java.util.Map<java.lang.String, java.lang.String>)).

labelEndpoints

public static void labelEndpoints(relationship r,
                                  Map<String,String> publishedColumnNames)

Labels each endpoint of r with the published name of the column its ids name (publishedColumnNames(gudusoft.gsqlparser.dlineage.dataflow.model.xml.dataflow)), so the label of an endpoint and the name of the column it points to never disagree: a merge that makes one column of several keeps one name, and the endpoints that named the others by their own spellings (a star link's folded display, the spelling of another reference) now carry it too. An endpoint whose ids name no published column keeps its label; routine calls are not column endpoints.

labelEndpoints

public static void labelEndpoints(dataflow flow)

labelEndpoints(gudusoft.gsqlparser.dlineage.dataflow.model.xml.relationship, java.util.Map<java.lang.String, java.lang.String>) for every relationship of flow, once its merge is done; then a relationship that now repeats another in every attribute but its id (exactKey(gudusoft.gsqlparser.dlineage.dataflow.model.xml.relationship): coordinates and the endpoints' ids included, so no evidence site and no edge is lost) is dropped, and the comment bindings that named it follow the one kept.

renameEndpoints

public static void renameEndpoints(List<relationship> relationships,
                                   Map<String,String> renamed)

identityOf

public static CanonKey identityOf(List<column> group)

The identity a group of columns stands for: the one its named columns carry, or null for a star or system group and for named columns that carry none.

carriedIdentities

public static DataflowUtility.SpilledIdentities carriedIdentities(dataflow df)

Captures the DataflowUtility.SpilledIdentities of a dataflow about to be serialized.

groupNames

public static DataflowUtility.GroupNames groupNames(List<table> group)

How a merge names the table it makes of group, in the group's order; see DataflowUtility.GroupNames.

groupIdentity

public static QualifiedCanonKey groupIdentity(List<table> merged)

The identity a model minted for any of the tables merged into one; they share one merge key.

carryGroupIdentity

public static void carryGroupIdentity(table merged,
                                      List<table> group,
                                      DataflowUtility.GroupNames chosen)

Gives a merged table the identity of the tables it replaces (the one a model minted), the marker of the table its names come from (whether those names are a reader's vouched SQL spelling), when those names are not ones a model minted the first minted names the group offers, and whether the group spans legacy folded names. The facts stay independent, so a later merge that includes the table chooses names as one merge of all the inputs would.

restoreCarriedIdentities

public static void restoreCarriedIdentities(dataflow df,
                                            DataflowUtility.SpilledIdentities identities)

readDataflowFromCsvMetadata

public static dataflow readDataflowFromCsvMetadata(String csvMetadata,
                                                   EDbVendor vendor)

mergeDataflowsAndCsv

public static Dataflow mergeDataflowsAndCsv(List<Pair<EDbVendor,dataflow>> pairs,
                                            String csvMetadata)

readDataflowFromCsvMetadata

public static dataflow readDataflowFromCsvMetadata(String csvMetadata)