Myujth

I am running MiNiFi in a Linux Box (gateway server) which is behind my company's firewall. My NiFi is running on an AWS EC2 cluster (running in standalone mode).

I am trying to send data from the Gateway to NiFi running in AWS EC2.

From gateway, I am able to telnet to EC2 node with the public DNS and the remote port which I have configured in the nifi.properties file

nifi.properties

# Site to Site properties
nifi.remote.input.host=ec2-xxx.us-east-2.compute.amazonaws.com
nifi.remote.input.secure=false
nifi.remote.input.socket.port=1026
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

Telnet connection from Gateway to NiFi

iot1@iothdp02:~/minifi/minifi-0.5.0/conf$ telnet ec2-xxx.us-east-2.compute.amazonaws.com 1026
Trying xx.xx.xx.xxx...
Connected to ec2-xxx.us-east-2.compute.amazonaws.com.
Escape character is '^]'.

The Public DNS is resolving to the correct Public IP of the EC2 node.

From the EC2 node, when I do nslookup on the Public DNS, it gives back the private IP.
From AWS Documentation: "The public IP address is mapped to the primary private IP address through network address translation (NAT). "

Hence, I am not adding the Public DNS and the Public IP in /etc/host file in the EC2 node.

From MiNiFi side, I am getting the below error:

minifi-app.log

iot1@iothdp02:~/minifi/minifi-0.5.0/logs$ cat minifi-app.log
2018-11-14 16:00:47,910 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2018-11-14 16:00:47,911 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2018-11-14 16:01:02,334 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@67207d8a checkpointed with 0 Records and 0 Swap Files in 20 milliseconds (Stop-the-world time = 6 milliseconds, Clear Edit Logs time = 4 millis), max Transaction ID -1
2018-11-14 16:02:47,911 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2018-11-14 16:02:47,912 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2018-11-14 16:03:02,354 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@67207d8a checkpointed with 0 Records and 0 Swap Files in 18 milliseconds (Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 5 millis), max Transaction ID -1
2018-11-14 16:03:10,636 WARN [Timer-Driven Process Thread-8] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from http://ec2-xxx.us-east-2.compute.amazonaws.com:9090/nifi-api due to java.net.UnknownHostException: ec2-xxx.us-east-2.compute.amazonaws.com: unknown error
2018-11-14 16:03:10,636 WARN [Timer-Driven Process Thread-8] o.apache.nifi.controller.FlowController Unable to communicate with remote instance RemoteProcessGroup[http://ec2-xxx.us-east-2.compute.amazonaws.com:9090/nifi] due to org.apache.nifi.controller.exception.CommunicationsException: org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI http://ec2-xxx.us-east-2.compute.amazonaws.com:9090/nifi due to: ec2-xxx.us-east-2.compute.amazonaws.com: unknown error
2018-11-14 16:04:47,912 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2018-11-14 16:04:47,912 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2018-11-14 16:05:02,380 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@67207d8a checkpointed with 0 Records and 0 Swap Files in 25 milliseconds (Stop-the-world time = 8 milliseconds, Clear Edit Logs time = 6 millis), max Transaction ID -1
2018-11-14 16:06:47,912 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2018-11-14 16:06:47,912 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2018-11-14 16:07:02,399 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@67207d8a checkpointed with

MiNiFi config.yml

MiNiFi Config Version: 3
Flow Controller:
 name: Gateway-IDS_v0.1
 comment: "1. ConsumeMQTT - MiNiFi will consume mqtt messages in gatewayn2. Remote
 Process Group will send messages to NiFi "
Core Properties:
 flow controller graceful shutdown period: 10 sec
 flow service write delay interval: 500 ms
 administrative yield duration: 30 sec
 bored yield duration: 10 millis
 max concurrent threads: 1
 variable registry properties: ''
FlowFile Repository:
 partitions: 256
 checkpoint interval: 2 mins
 always sync: false
 Swap:
 threshold: 20000
 in period: 5 sec
 in threads: 1
 out period: 5 sec
 out threads: 4
Content Repository:
 content claim max appendable size: 10 MB
 content claim max flow files: 100
 always sync: false
Provenance Repository:
 provenance rollover time: 1 min
 implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
Component Status Repository:
 buffer size: 1440
 snapshot frequency: 1 min
Security Properties:
 keystore: ''
 keystore type: ''
 keystore password: ''
 key password: ''
 truststore: ''
 truststore type: ''
 truststore password: ''
 ssl protocol: ''
 Sensitive Props:
 key:
 algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
 provider: BC
Processors:
- id: 6396f40f-118f-33f4-0000-000000000000
 name: ConsumeMQTT
 class: org.apache.nifi.processors.mqtt.ConsumeMQTT
 max concurrent tasks: 1
 scheduling strategy: TIMER_DRIVEN
 scheduling period: 0 sec
 penalization period: 30 sec
 yield period: 1 sec
 run duration nanos: 0
 auto-terminated relationships list: 
 Properties:
 Broker URI: tcp://localhost:1883
 Client ID: nifi
 Connection Timeout (seconds): '30'
 Keep Alive Interval (seconds): '60'
 Last Will Message:
 Last Will QoS Level:
 Last Will Retain:
 Last Will Topic:
 MQTT Specification Version: '0'
 Max Queue Size: '10'
 Password:
 Quality of Service(QoS): '0'
 SSL Context Service:
 Session state: 'true'
 Topic Filter: MQTT
 Username:
Controller Services: 
Process Groups: 
Input Ports: 
Output Ports: 
Funnels: 
Connections:
- id: f0007aa3-cf32-3593-0000-000000000000
 name: ConsumeMQTT/Message/85ebf198-0166-1000-5592-476a7ba47d2e
 source id: 6396f40f-118f-33f4-0000-000000000000
 source relationship names:
 - Message
 destination id: 85ebf198-0166-1000-5592-476a7ba47d2e
 max work queue size: 10000
 max work queue data size: 1 GB
 flowfile expiration: 0 sec
 queue prioritizer class: ''
Remote Process Groups:
- id: c00d3132-375b-323f-0000-000000000000
 name: ''
 url: http://ec2-xxx.us-east-2.compute.amazonaws.com:9090
 comment: ''
 timeout: 30 sec
 yield period: 10 sec
 transport protocol: RAW
 proxy host: ''
 proxy port: ''
 proxy user: ''
 proxy password: ''
 local network interface: ''
 Input Ports:
 - id: 85ebf198-0166-1000-5592-476a7ba47d2e
 name: From MiNiFi
 comment: ''
 max concurrent tasks: 1
 use compression: false
 Properties: 
 Port: 1026 
 Host Name: ec2-xxx.us-east-2.compute.amazonaws.com
 Output Ports: 
NiFi Properties Overrides: 

Any pointers on how to troubleshoot this issue?