Job work which not attract GST should not come in itc-04

I want to ask little things about ITC-04.

I am sending my plain saree for job work, 

Job worker is unregistered. So he did not charged any gst on his bill of job work charges. 

Now there is no GST paid by me so no matter to take input of GST, if no matter of input then it does not require to file ITC-04 for those type of JOB Work. 

Kindly clear me, am I right....??