Extract unique ip address from log files and write it to a new file, in windows using powershell

We have a log file, which has ip addresses in it. And, we have been asked to extract all the Ip address, without any duplicates. And, we are on windows system, so, how are we going to do it? You can watch me do it on youtube. [embed]https://youtu.be/Ugv-uUM9-zg[/embed] First we need to read the content of that file. Replace the .\TestFiles\Linux_2k.log path, with path to your file.

Get-Content -Path .\TestFiles\Linux_2k.log

Second, we need to pattern match the IP addresses in the file. For that, we will use Select-String for that. Now, our command looks like this. "(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" is the pattern to match an ip addresses, I have surrouned the pattern in parentheses to make it a group.

Get-Content -Path .\TestFiles\Linux_2k.log 
| select-string -Pattern "(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"

Third, we need to extract Matcher-Group-Value to get the ip address. As, Select-String command returns an instance of MatchInfo class.

Get-Content -Path .\TestFiles\Linux_2k.log 
| select-string -Pattern "(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" 
| Select-Object -Property @{ Label="Ip Address"; expression={ $PSItem.Matches.Groups[1].Value } }

The above command will return all the ip addresses in the file. It also returns the duplicates, currently it is returning 1245 ips. But, we don't want the duplicates.

To remove duplicates, we have to pass -Unique argument to the Select-Object command.

Get-Content -Path .\TestFiles\Linux_2k.log 
| select-string -Pattern "(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" 
| Select-Object -Property @{ Label="Ip Address"; expression={ $PSItem.Matches.Groups[1].Value } } -Unique

The above command will give us all the unique ips. Now total unique ips returned are 68, the duplicates are removed.

But, we need these ips in a file. How are we going to do that? We can use Out-File command to write these ips in a file.

Get-Content -Path .\TestFiles\Linux_2k.log 
| select-string -Pattern "(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" 
| Select-Object -Property @{ Label="Ip Address"; expression={ $PSItem.Matches.Groups[1].Value } } -Unique 
| Out-File -FilePath UniqueIpAddresses.txt;

The list of unique ip address will get stored in UniqueIpAddresses.txt file. Which you can email, send over message or it can be printed. That means it can be shared.