REST provenance/lineage API

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

REST provenance/lineage API

philippe.gibert

Hello Matt, I have the same problem with the provenance/lineage REST call

 

curl 'http://localhost:8080/nifi-api/provenance/lineage/…………..

 

very difficult for me (sorry L ) to  find the right  json  body to send in the POST call .

if you can give an example again

 

Philippe

Best regards

-------------------------------------------------------------------------------------------------------------

Thanks a lot Matt

It works nicely !

 

 

From: Matt Gilman [[hidden email]]
Sent: mardi 15 novembre 2016 15:08
To: [hidden email]
Subject: Re: REST provenance api search options

 

Philippe,

 

Here's an example command for initiating a provenance search:

 

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"FlowFileUUID":"<flowfile-uuid>","Filename":"<name of flowfile>","ProcessorID":"<component-uuid>"}}}}' --compressed

 

The available searchable fields are defined in your nifi.properties file under the following properties:

 

nifi.provenance.repository.indexed.fields

nifi.provenance.repository.indexed.attributes

 

Because the searches can be long running they are performed asynchronously. These means that the curl command above creates the search request but does not wait for it to complete. Instead, you'll need to get the uuid to the search request to continue to GET it until the search completes. Once completed, you should DELETE the search request. Open up the Dev Tools in your browser to seen this sequence of requests in action.

 

Let me know if you have any more questions. Thanks.

 

Matt

 

 

On Tue, Nov 15, 2016 at 2:47 AM, <[hidden email]> wrote:

Hello, 

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my processor ie (ProcessorID ) but it’s not very clear for me how to fill the searchableFields. 

Is something  as the following to be right  ? and in this case where do i put the ProcessorID? in the field or  in the Id ? and what for label?

-----------------------------------------------

curl -i -X PUT -H 'Content-Type: application/json' -d '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options

--------------------------------------------------

​ please can you clarify ? with an example ?

Best regards 

phil

 

 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: REST provenance/lineage API

Matt Gilman
Phil,

The lineage endpoints follow the same asynchronous model that the provenance endpoints do. Here is an example curl command to initiate a lineage search request:

curl 'https://localhost:8443/nifi-api/provenance/lineage' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"lineage":{"request":{"lineageRequestType":"FLOWFILE","uuid":"05f547fe-e15d-4186-a645-fbed6621aee1","eventId":"0"}}}' --compressed

The contents of this request comes from the details of a selected provenance event. I'd suggest using your browser's Dev Tools to watch the requests in action while you are searching provenance and viewing lineage graphs. 

Let me know if you have any other questions. Thanks!

Matt

On Wed, Nov 16, 2016 at 11:15 AM, <[hidden email]> wrote:

Hello Matt, I have the same problem with the provenance/lineage REST call

 

curl 'http://localhost:8080/nifi-api/provenance/lineage/…………..

 

very difficult for me (sorry L ) to  find the right  json  body to send in the POST call .

if you can give an example again

 

Philippe

Best regards

-------------------------------------------------------------------------------------------------------------

Thanks a lot Matt

It works nicely !

 

 

From: Matt Gilman [[hidden email]]
Sent: mardi 15 novembre 2016 15:08
To: [hidden email]
Subject: Re: REST provenance api search options

 

Philippe,

 

Here's an example command for initiating a provenance search:

 

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"FlowFileUUID":"<flowfile-uuid>","Filename":"<name of flowfile>","ProcessorID":"<component-uuid>"}}}}' --compressed

 

The available searchable fields are defined in your nifi.properties file under the following properties:

 

nifi.provenance.repository.indexed.fields

nifi.provenance.repository.indexed.attributes

 

Because the searches can be long running they are performed asynchronously. These means that the curl command above creates the search request but does not wait for it to complete. Instead, you'll need to get the uuid to the search request to continue to GET it until the search completes. Once completed, you should DELETE the search request. Open up the Dev Tools in your browser to seen this sequence of requests in action.

 

Let me know if you have any more questions. Thanks.

 

Matt

 

 

On Tue, Nov 15, 2016 at 2:47 AM, <[hidden email]> wrote:

Hello, 

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my processor ie (ProcessorID ) but it’s not very clear for me how to fill the searchableFields. 

Is something  as the following to be right  ? and in this case where do i put the ProcessorID? in the field or  in the Id ? and what for label?

-----------------------------------------------

curl -i -X PUT -H 'Content-Type: application/json' -d '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options

--------------------------------------------------

​ please can you clarify ? with an example ?

Best regards 

phil

 

 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Reply | Threaded
Open this post in threaded view
|

RE: REST provenance/lineage API

philippe.gibert
In reply to this post by philippe.gibert

Hello matt

Almost perfect !

I call nifi REST api with the following JSON body in the post ( just indicating the flow uuid)

----------

{ "lineage":{  "request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc" } } }

--------------

And  results are records like this but I would expect  the ProcessorID ( componentID?) and  Processor Name involved in each record ? is it possible ?

Philippe

 

{

"id":"322",

"flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",

"type":"EVENT",

"eventType":"RECEIVE",

"millis":1479375558360,

"timestamp":"11/17/2016 09:39:18.360 UTC”

}

 

 

 

From: GIBERT Philippe IMT/OLPS
Sent: mercredi 16 novembre 2016 17:15
To: [hidden email]
Subject: REST provenance/lineage API

 

Hello Matt, I have the same problem with the provenance/lineage REST call

 

curl 'http://localhost:8080/nifi-api/provenance/lineage/…………..

 

very difficult for me (sorry L ) to  find the right  json  body to send in the POST call .

if you can give an example again

 

Philippe

Best regards

 

-------------------------------------------------------------------------------------------------------------

Thanks a lot Matt

It works nicely !

 

 

From: Matt Gilman [[hidden email]]
Sent: mardi 15 novembre 2016 15:08
To: [hidden email]
Subject: Re: REST provenance api search options

 

Philippe,

 

Here's an example command for initiating a provenance search:

 

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"FlowFileUUID":"<flowfile-uuid>","Filename":"<name of flowfile>","ProcessorID":"<component-uuid>"}}}}' --compressed

 

The available searchable fields are defined in your nifi.properties file under the following properties:

 

nifi.provenance.repository.indexed.fields

nifi.provenance.repository.indexed.attributes

 

Because the searches can be long running they are performed asynchronously. These means that the curl command above creates the search request but does not wait for it to complete. Instead, you'll need to get the uuid to the search request to continue to GET it until the search completes. Once completed, you should DELETE the search request. Open up the Dev Tools in your browser to seen this sequence of requests in action.

 

Let me know if you have any more questions. Thanks.

 

Matt

 

 

On Tue, Nov 15, 2016 at 2:47 AM, <[hidden email]> wrote:

Hello, 

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my processor ie (ProcessorID ) but it’s not very clear for me how to fill the searchableFields. 

Is something  as the following to be right  ? and in this case where do i put the ProcessorID? in the field or  in the Id ? and what for label?

-----------------------------------------------

curl -i -X PUT -H 'Content-Type: application/json' -d '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options

--------------------------------------------------

​ please can you clarify ? with an example ?

Best regards 

phil

 

 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: REST provenance/lineage API

Matt Gilman
Phil,

Those details are available in the provenance event which can be accessed through this endpoint:
GET /nifi-api/provenance-events/<event-id>
If your instance is clustered however, you'll also need to include clusterNodeId as a query parameter.

Matt

On Thu, Nov 17, 2016 at 8:46 AM, <[hidden email]> wrote:

Hello matt

Almost perfect !

I call nifi REST api with the following JSON body in the post ( just indicating the flow uuid)

----------

{ "lineage":{  "request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc" } } }

--------------

And  results are records like this but I would expect  the ProcessorID ( componentID?) and  Processor Name involved in each record ? is it possible ?

Philippe

 

{

"id":"322",

"flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",

"type":"EVENT",

"eventType":"RECEIVE",

"millis":1479375558360,

"timestamp":"11/17/2016 09:39:18.360 UTC”

}

 

 

 

From: GIBERT Philippe IMT/OLPS
Sent: mercredi 16 novembre 2016 17:15
To: [hidden email]
Subject: REST provenance/lineage API

 

Hello Matt, I have the same problem with the provenance/lineage REST call

 

curl 'http://localhost:8080/nifi-api/provenance/lineage/…………..

 

very difficult for me (sorry L ) to  find the right  json  body to send in the POST call .

if you can give an example again

 

Philippe

Best regards

 

-------------------------------------------------------------------------------------------------------------

Thanks a lot Matt

It works nicely !

 

 

From: Matt Gilman [[hidden email]]
Sent: mardi 15 novembre 2016 15:08
To: [hidden email]
Subject: Re: REST provenance api search options

 

Philippe,

 

Here's an example command for initiating a provenance search:

 

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"FlowFileUUID":"<flowfile-uuid>","Filename":"<name of flowfile>","ProcessorID":"<component-uuid>"}}}}' --compressed

 

The available searchable fields are defined in your nifi.properties file under the following properties:

 

nifi.provenance.repository.indexed.fields

nifi.provenance.repository.indexed.attributes

 

Because the searches can be long running they are performed asynchronously. These means that the curl command above creates the search request but does not wait for it to complete. Instead, you'll need to get the uuid to the search request to continue to GET it until the search completes. Once completed, you should DELETE the search request. Open up the Dev Tools in your browser to seen this sequence of requests in action.

 

Let me know if you have any more questions. Thanks.

 

Matt

 

 

On Tue, Nov 15, 2016 at 2:47 AM, <[hidden email]> wrote:

Hello, 

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my processor ie (ProcessorID ) but it’s not very clear for me how to fill the searchableFields. 

Is something  as the following to be right  ? and in this case where do i put the ProcessorID? in the field or  in the Id ? and what for label?

-----------------------------------------------

curl -i -X PUT -H 'Content-Type: application/json' -d '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options

--------------------------------------------------

​ please can you clarify ? with an example ?

Best regards 

phil

 

 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Reply | Threaded
Open this post in threaded view
|

RE: REST provenance/lineage API

philippe.gibert

 

Thanks , very simple at the end J !

 

Last question  , If I can ..

Initially I have a  process group named  PG1 with some processors.

I want to Execute the DFlow  located in PG1

and then  with and external program  call via REST lineage/provenance  API to get the complete  report about the flow.

My question is :

Is there a way to start  with the ProcessgroupName ( ie PG1)   and get the  flow_uid  executed in the process Group with 1 REST API ?

 

Best regards

Phil

 

 

From: Matt Gilman [mailto:[hidden email]]
Sent: jeudi 17 novembre 2016 15:25
To: [hidden email]
Subject: Re: REST provenance/lineage API

 

Phil,

 

Those details are available in the provenance event which can be accessed through this endpoint:

GET /nifi-api/provenance-events/<event-id>

If your instance is clustered however, you'll also need to include clusterNodeId as a query parameter.

 

Matt

 

On Thu, Nov 17, 2016 at 8:46 AM, <[hidden email]> wrote:

Hello matt

Almost perfect !

I call nifi REST api with the following JSON body in the post ( just indicating the flow uuid)

----------

{ "lineage":{  "request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc" } } }

--------------

And  results are records like this but I would expect  the ProcessorID ( componentID?) and  Processor Name involved in each record ? is it possible ?

Philippe

 

{

"id":"322",

"flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",

"type":"EVENT",

"eventType":"RECEIVE",

"millis":1479375558360,

"timestamp":"11/17/2016 09:39:18.360 UTC”

}

 

 

 

From: GIBERT Philippe IMT/OLPS
Sent: mercredi 16 novembre 2016 17:15
To: [hidden email]
Subject: REST provenance/lineage API

 

Hello Matt, I have the same problem with the provenance/lineage REST call

 

curl 'http://localhost:8080/nifi-api/provenance/lineage/…………..

 

very difficult for me (sorry L ) to  find the right  json  body to send in the POST call .

if you can give an example again

 

Philippe

Best regards

 

-------------------------------------------------------------------------------------------------------------

Thanks a lot Matt

It works nicely !

 

 

From: Matt Gilman [[hidden email]]
Sent: mardi 15 novembre 2016 15:08
To: [hidden email]
Subject: Re: REST provenance api search options

 

Philippe,

 

Here's an example command for initiating a provenance search:

 

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"FlowFileUUID":"<flowfile-uuid>","Filename":"<name of flowfile>","ProcessorID":"<component-uuid>"}}}}' --compressed

 

The available searchable fields are defined in your nifi.properties file under the following properties:

 

nifi.provenance.repository.indexed.fields

nifi.provenance.repository.indexed.attributes

 

Because the searches can be long running they are performed asynchronously. These means that the curl command above creates the search request but does not wait for it to complete. Instead, you'll need to get the uuid to the search request to continue to GET it until the search completes. Once completed, you should DELETE the search request. Open up the Dev Tools in your browser to seen this sequence of requests in action.

 

Let me know if you have any more questions. Thanks.

 

Matt

 

 

On Tue, Nov 15, 2016 at 2:47 AM, <[hidden email]> wrote:

Hello, 

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my processor ie (ProcessorID ) but it’s not very clear for me how to fill the searchableFields. 

Is something  as the following to be right  ? and in this case where do i put the ProcessorID? in the field or  in the Id ? and what for label?

-----------------------------------------------

curl -i -X PUT -H 'Content-Type: application/json' -d '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options

--------------------------------------------------

​ please can you clarify ? with an example ?

Best regards 

phil

 

 

_________________________________________________________________________________________________________________________
 
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
 
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.