{"id":1221,"date":"2023-05-26T13:56:17","date_gmt":"2023-05-26T13:56:17","guid":{"rendered":"http:\/\/stage.wizzdev.pl\/?p=1221"},"modified":"2023-05-26T14:01:50","modified_gmt":"2023-05-26T14:01:50","slug":"fpga-data-transfer-demo-5","status":"publish","type":"post","link":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/","title":{"rendered":"FPGA Data Transfer demo #5"},"content":{"rendered":"
<\/div>\n

Reading data from FPGA<\/h2>\n

The FPGADevice class uses Opal Kelly Python API to:
\n– load bitfile and configure the board
\nusing:<\/p>\n

okCFrontPanel.ConfigureFPGA(bit_file_path)<\/pre>\n
– control data generation
\nby integrating:<\/p>\n
okCFrontPanel.SetWireInValue(endpoint_addr, value)\r\nokCFrontPanel.ActivateTriggerIn(endpoint_addr, bit)\r\n<\/pre>\n
– receive data generated by the module<\/p>\n
okCFrontPanel.ReadFromBlockPipeOut(endpoint_addr, block_size, data)\r\n<\/pre>\n
Current transfer (data) length setting is 1024 bytes. This value should be adjusted to meet the end application needs. For example, it can be increased to enable proper throughput with higher data generation rate. The corresponding parameter on the FPGA side should be also changed then.
\nReceived data is then pushed to further processing.<\/p>\n
Data unpacker<\/h2>\n
Received buffers are then fed to the “Data unpacker”‘s queue. The “Data unpacker” operates in a separate thread and process buffers by unpacking them from thepacket structure (described in post#3<\/a> ).
\nUnpacked data is then sent to the Data Manager’s queue.<\/p>\n
Data manager<\/h2>\n
The main (and only) job of this module is to feed the buffers from its queue to the queues of all further processors connected to it. Currently, these are: plot data source<\/strong> and hdf5 writer<\/strong>.<\/p>\n
Plotting<\/h2>\n
Plot data source stores data, and returns them on request. Currently acquiring the data for specified channel and time span is implemented. It is added as a data handle for Plotting widget of the Gui application. pyqtgraph python library is used to display the data.<\/p>\n
Writing to HDF<\/h2>\n
Why HDF5?
\nHDF5 is the standardized format used widely in academic and industry fields.
\nIt has no data size limit or maximum allowed dimensions so it’s perfect for big data such as multiple or high-frequency time series.
\nIt organises the data into datasets (multidimensional arrays of specified datatype) which can be gathered into different groups.
\nAlso, dataset chunking is supported which leads to better performance and possible compression.
\nAdditionally, every complex data type within HDF5 file may have some defined attributes.<\/p>\n
In our case, the dataset dimensions are NUMBER_OF_SOURCES (constant value for a single dataset) by NUMBER_OF_TIME_STAMPS (which is expanded every time new data appear). Two attributes: “start_time” and “sampling_interval” – are there to enable time vector reconstruction.<\/p>\n
Reading from HDF<\/h2>\n
You can open the resulting files using the popular HDFView (version >= 3.0) or ViTables, but we found that HDF5View’s plotting abilities are rather limited, and ViTables does not provide one. Knowing that and having the online data visualisation already implemented we wrote our own module for viewing data stored in HDF5 files.
\nFor the purpose of this demo, a whole .h5 file is read into a PlotData object, therefore, the user is able to explore the whole timespan of the selected channel. It does not lead to any performance threats provided that we are dealing with a relatively small amount of data.<\/p>\n
[code language=”python”]
\nclass HDF5FileReader:
\ndef __init__(self, file_path):
\nself.file = h5py.File(file_path, ‘r’)<\/p>\n
group_name = list(self.file.keys())[0]
\nself.group = self.file[group_name]
\ndata_set_name = list(self.group.keys())[0]
\nself.data_set = self.group[data_set_name]<\/p>\n
self.number_of_sources = self.data_set.shape[1]
\nself.time_span = self.data_set.shape[0]<\/p>\n
self.start_time = self.data_set.attrs[‘start_time’]
\nself.sampling_interval = self.data_set.attrs[‘sampling_interval’]
\nself.sampling_rate = int(1\/self.sampling_interval)
\n(…)<\/p>\n
[\/code]<\/p>\n
The above code snipped shows how to read the data from HDF5 file using h5py python module.<\/p>\n","protected":false},"excerpt":{"rendered":"
Reading data from FPGA The FPGADevice class uses Opal Kelly Python API to: – load bitfile and configure the board using: okCFrontPanel.ConfigureFPGA(bit_file_path) – control data generation by integrating: okCFrontPanel.SetWireInValue(endpoint_addr, value) okCFrontPanel.ActivateTriggerIn(endpoint_addr, \u00a0[…]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":""},"categories":[7],"tags":[],"acf":[],"yoast_head":"\nFPGA Data Transfer demo #5 - WizzDev<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"FPGA Data Transfer demo #5 - WizzDev\" \/>\n<meta property=\"og:description\" content=\"Reading data from FPGA The FPGADevice class uses Opal Kelly Python API to: – load bitfile and configure the board using: okCFrontPanel.ConfigureFPGA(bit_file_path) – control data generation by integrating: okCFrontPanel.SetWireInValue(endpoint_addr, value) okCFrontPanel.ActivateTriggerIn(endpoint_addr, \u00a0[...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/\" \/>\n<meta property=\"og:site_name\" content=\"WizzDev\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-26T13:56:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-26T14:01:50+00:00\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/\"},\"author\":{\"name\":\"admin\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/#\/schema\/person\/a82672ac0e310ffa148007ec229c09cf\"},\"headline\":\"FPGA Data Transfer demo #5\",\"datePublished\":\"2023-05-26T13:56:17+00:00\",\"dateModified\":\"2023-05-26T14:01:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/\"},\"wordCount\":556,\"publisher\":{\"@id\":\"https:\/\/stage2.wizzdev.pl\/#organization\"},\"articleSection\":[\"FPGA projects\"],\"inLanguage\":\"en\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/\",\"url\":\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/\",\"name\":\"FPGA Data Transfer demo #5 - WizzDev\",\"isPartOf\":{\"@id\":\"https:\/\/stage2.wizzdev.pl\/#website\"},\"datePublished\":\"2023-05-26T13:56:17+00:00\",\"dateModified\":\"2023-05-26T14:01:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/#breadcrumb\"},\"inLanguage\":\"en\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/stage2.wizzdev.pl\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"FPGA Data Transfer demo #5\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/#website\",\"url\":\"https:\/\/stage2.wizzdev.pl\/\",\"name\":\"WizzDev\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/stage2.wizzdev.pl\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/stage2.wizzdev.pl\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/#organization\",\"name\":\"WizzDev\",\"url\":\"https:\/\/stage2.wizzdev.pl\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/stage2.wizzdev.pl\/wp-content\/uploads\/2022\/02\/Logo-WizzDev-wersja-1-PNG-granat-niebieski-1-1.jpg\",\"contentUrl\":\"https:\/\/stage2.wizzdev.pl\/wp-content\/uploads\/2022\/02\/Logo-WizzDev-wersja-1-PNG-granat-niebieski-1-1.jpg\",\"width\":179,\"height\":47,\"caption\":\"WizzDev\"},\"image\":{\"@id\":\"https:\/\/stage2.wizzdev.pl\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/twitter.com\/wizz_dev\",\"https:\/\/www.upwork.com\/ag\/wizzdev\/\",\"https:\/\/github.com\/wizzdev-pl\",\"https:\/\/linkedin.com\/company\/wizzdev\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/#\/schema\/person\/a82672ac0e310ffa148007ec229c09cf\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en\",\"@id\":\"https:\/\/stage2.wizzdev.pl\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a040dcbe510bab2c1867895ac03435c7?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a040dcbe510bab2c1867895ac03435c7?s=96&d=mm&r=g\",\"caption\":\"admin\"},\"sameAs\":[\"https:\/\/stage2.wizzdev.pl\"],\"url\":\"https:\/\/stage2.wizzdev.pl\/blog\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"FPGA Data Transfer demo #5 - WizzDev","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/","og_locale":"en_US","og_type":"article","og_title":"FPGA Data Transfer demo #5 - WizzDev","og_description":"Reading data from FPGA The FPGADevice class uses Opal Kelly Python API to: – load bitfile and configure the board using: okCFrontPanel.ConfigureFPGA(bit_file_path) – control data generation by integrating: okCFrontPanel.SetWireInValue(endpoint_addr, value) okCFrontPanel.ActivateTriggerIn(endpoint_addr, \u00a0[...]","og_url":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/","og_site_name":"WizzDev","article_published_time":"2023-05-26T13:56:17+00:00","article_modified_time":"2023-05-26T14:01:50+00:00","author":"admin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"admin","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/#article","isPartOf":{"@id":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/"},"author":{"name":"admin","@id":"https:\/\/stage2.wizzdev.pl\/#\/schema\/person\/a82672ac0e310ffa148007ec229c09cf"},"headline":"FPGA Data Transfer demo #5","datePublished":"2023-05-26T13:56:17+00:00","dateModified":"2023-05-26T14:01:50+00:00","mainEntityOfPage":{"@id":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/"},"wordCount":556,"publisher":{"@id":"https:\/\/stage2.wizzdev.pl\/#organization"},"articleSection":["FPGA projects"],"inLanguage":"en"},{"@type":"WebPage","@id":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/","url":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/","name":"FPGA Data Transfer demo #5 - WizzDev","isPartOf":{"@id":"https:\/\/stage2.wizzdev.pl\/#website"},"datePublished":"2023-05-26T13:56:17+00:00","dateModified":"2023-05-26T14:01:50+00:00","breadcrumb":{"@id":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/#breadcrumb"},"inLanguage":"en","potentialAction":[{"@type":"ReadAction","target":["https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/stage2.wizzdev.pl\/blog\/fpga-data-transfer-demo-5\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/stage2.wizzdev.pl\/"},{"@type":"ListItem","position":2,"name":"FPGA Data Transfer demo #5"}]},{"@type":"WebSite","@id":"https:\/\/stage2.wizzdev.pl\/#website","url":"https:\/\/stage2.wizzdev.pl\/","name":"WizzDev","description":"","publisher":{"@id":"https:\/\/stage2.wizzdev.pl\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/stage2.wizzdev.pl\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en"},{"@type":"Organization","@id":"https:\/\/stage2.wizzdev.pl\/#organization","name":"WizzDev","url":"https:\/\/stage2.wizzdev.pl\/","logo":{"@type":"ImageObject","inLanguage":"en","@id":"https:\/\/stage2.wizzdev.pl\/#\/schema\/logo\/image\/","url":"https:\/\/stage2.wizzdev.pl\/wp-content\/uploads\/2022\/02\/Logo-WizzDev-wersja-1-PNG-granat-niebieski-1-1.jpg","contentUrl":"https:\/\/stage2.wizzdev.pl\/wp-content\/uploads\/2022\/02\/Logo-WizzDev-wersja-1-PNG-granat-niebieski-1-1.jpg","width":179,"height":47,"caption":"WizzDev"},"image":{"@id":"https:\/\/stage2.wizzdev.pl\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/twitter.com\/wizz_dev","https:\/\/www.upwork.com\/ag\/wizzdev\/","https:\/\/github.com\/wizzdev-pl","https:\/\/linkedin.com\/company\/wizzdev"]},{"@type":"Person","@id":"https:\/\/stage2.wizzdev.pl\/#\/schema\/person\/a82672ac0e310ffa148007ec229c09cf","name":"admin","image":{"@type":"ImageObject","inLanguage":"en","@id":"https:\/\/stage2.wizzdev.pl\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a040dcbe510bab2c1867895ac03435c7?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a040dcbe510bab2c1867895ac03435c7?s=96&d=mm&r=g","caption":"admin"},"sameAs":["https:\/\/stage2.wizzdev.pl"],"url":"https:\/\/stage2.wizzdev.pl\/blog\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/posts\/1221"}],"collection":[{"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/comments?post=1221"}],"version-history":[{"count":3,"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/posts\/1221\/revisions"}],"predecessor-version":[{"id":4780,"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/posts\/1221\/revisions\/4780"}],"wp:attachment":[{"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/media?parent=1221"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/categories?post=1221"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stage2.wizzdev.pl\/wp-json\/wp\/v2\/tags?post=1221"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}