Mass Ingestion

Back Next

Python transformation

A Python transformation runs the Python script to transform incoming data from a streaming source.

A Python transformation processes binary, JSON, and XML data. The Python transformation uses the two variables, inputData and outputData for storing the incoming data and outgoing data.

Incoming data of XML or JSON message formats are stored as string in the inputData variable and the outgoing data of XML or JSON message formats are stored as string in the outputData. Incoming data of binary message format are stored as

numpy.ndarray

in the inputData variable. Outgoing data of binary message format are stored as

bytearray

in the outputData variable.

Binary data in the inputData variable are encoded as ASCII characters. You must decode the data accordingly. Also ensure that the Python transformation script handles any non-ASCII characters present in the inputData variable.

Before using a Python transformation, create a directory, Python home to install Python. After installing Python in the Python home directory, ensure to install the third-party libraries, NumPy and Jep (Java Embedded Python) in the same directory as Python home.

In one Secure Agent, you cannot use two different versions of Python to run the same Python transformation.

Sample Python scripts for JSON

import json
temp=json.loads(inputData)
temp["name"]="Mr "+temp["name"]
outputData=json.dumps(temp)
###################################################################
inputData: { "name":"John", "age":30, "city":"New York"}
outputData: { "name":"Mr John", "age":30, "city":"New York"}

Sample Python scripts for binary

temp = ''.join(str(chr(c)) for c in inputData)
temp += " - this is edited again text"
outputData = bytearray(temp, 'utf-8')
###################################################################
inputData: Sample text
outputData: Sample text - this is edited again text

Sample Python scripts for XML

import xml.etree.ElementTree as ET
myroot = ET.fromstring(inputData)
for x in myroot:
     if x.tag=="body":
         x.tag="Msg" 
xmlstr = ET.tostring(myroot)
outputData=xmlstr.decode('utf-8')
###################################################################
inputData: <note><to>You</to><from>Me</from><heading>Message</heading><body>Happy Coding</body></note>
outputData: <note><to>You</to><from>Me</from><heading>Message</heading><Msg>Happy Coding</Msg></note>

Rename Saved Search

Table of Contents

Mass Ingestion

Mass Ingestion

Python transformation

Python transformation

Sample Python scripts for JSON

Sample Python scripts for binary

Sample Python scripts for XML