Client IPSEC VPN Configurations on a Cisco ASA 5500

Below I've put together a rough configuration of what I did to build my own home Remote Access VPN on a Cisco ASA5505.  I really put this up for my own personal reference, but feel free to post any questions.  Most of this may seem vague if you're unfamiliar with the ASA platform/configurations.



Basic Crypto configurations required for my example:

Declare the P2 encryption to be used:

crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 

Crypto Map is named "outside_map" and applied to the outside interface:

crypto map outside_map interface outside

I also used ikev1:

crypto ikev1 enable outside

Phase 1 configuration:

crypto ikev1 policy 65535
 authentication pre-share
 encryption 3des
 hash sha   
 group 2    
 lifetime 86400

Have a local user account to test with:

username cisco password cisco privilege 15



Beginning of configuration:

Create the Objects that reference your internal networks and User IP Pool:

object network obj-vpn-users
 subnet 192.168.99.0 255.255.255.248

object network obj-internal
 subnet 192.168.1.0 255.255.255.0



Create the ACLs that will be applied to the user at time of VPN authentication:

access-list ACL_USER extended permit ip 192.168.99.0 255.255.255.248 192.168.1.0 255.255.255.0


Create the Split Tunnel ACLs:

access-list ACL_Split-tunnel-USER standard permit 192.168.1.0 255.255.255.0


Create the IP Pool for VPN assignment:

ip local pool POOL_USER 192.168.99.2-192.168.99.6 mask 255.255.255.248


Create the No NAT policy for inside traffic to reach the VPN user (make sure that this is in your configuration before any dynamic PAT policy:

nat (inside,outside) source static obj-internal obj-internal destination static obj-vpn-users obj-vpn-users description USER_VPN


Create a dynamic map to support the VPN connection from any location and set the transform-set and reverse-route:

crypto dynamic-map VPN-USER 1 set ikev1 transform-set ESP-3DES-SHA
crypto dynamic-map VPN-USER 1 set reverse-route

crypto map outside_map 999 ipsec-isakmp dynamic VPN-USER


Configure the Group Policy:


group-policy GP_USER internal
group-policy GP_USER attributes
 vpn-filter value ACL_USER
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value ACL_Split-tunnel-USER

Finally Create your tunnel group specifying the group policy and pre shared key:


tunnel-group VPN-USER type remote-access
tunnel-group VPN-USER general-attributes
 address-pool POOL_USER
 default-group-policy GP_USER
tunnel-group VPN-USER ipsec-attributes
 ikev1 pre-shared-key cisco123

Port Redirection in Linux

Here's a simple Port Redirection example in Linux, I use this a lot in "work around" solutions, because it's essentially a static PAT without the need for network equipment.

From your Linux host the command is:

ssh localhost -L <LocalListeningIP>:<LocalListeningPort>:<RemoteServerIP>:<RemoteServerPort> -N

- localhost : This simply initiates the ssh session to the localhost for the port redirection to take affect.
- "-L": The switch to trigger the port redirection and accept the following variables
- LocalListeningIP : The IP address that you want to listen on.  This can be 0.0.0.0 if you want to listen to all local IPs on the system
- LocalListeningPort : The TCP Port that you want to listen to.  This can be any port.  Your internal systems will target this to reach the remote server's port.
- RemoteServerIP : The remote server's IP address that your internal systems will be targeting and redirected to.
- RemoteServerPort : The remote server's listening Port that your internal systems will need to reach.
- "-N" : This tells the system not to execute any commands once the ssh session is established.

Best Practice for Using BGP Attributes to Modify Routing

       The BGP routing protocol has been known as one of the best routing protocols when there's any type of complex or even simple routing manipulation it has to occur. In many ways I like to consider it almost a kind of "all-terrain" routing protocol.   It functions great anywhere you go!  90% of the time it serves as a great IGP.  It was built to stand alone as an EGP, but when it comes to it's functionality, it really does do more than OSPF and EIGRP combined.

       In this post I'm going to briefly go through the BGP attributes and note the ones that are most commonly used when peering 2 EBGP neighbors for a multihomed setup as diagramed below.

These are the 8 BGP attributes as listed by Cisco.

W   Weight (Highest)
L   Local_Pref (Highest)
O   Originate (local originate)
AS  As_Path (shortest)
O   Origin Code (IGP < EGP < Incomplete)
M   MED (lowest)
P   Paths (External Paths preferred Over Internal)
R   Router ID (lowest)

Weight:
The Weight attribute only applies to Cisco routers and is local to the router, so other BGP peers along the path will not know about the attribute.  Since it's Cisco proprietary it's also not a popular selection as a route modifier.  The best bet is to stay away from it in a production environment.

Local Preference:
The local preference attribute is popular for route modifications since it is propagated within the entire AS.  It's definitely an attribute that is excellent to use when routes need to be followed deep within your AS.  This is a very popular attribute to modify for EBGP and IBGP routes and is highly recommended.

Originate:
The Originate attribute looks at how a route was learned locally.  Since this is not advertised within the local AS or to other AS's, it's best not to rely on this for route manipulation.

AS Path:
The AS Path attribute is one of my favorites because it is easy to set and follow.  It also gives you the ability to create filters within your network to avoid routes based upon an AS within the path.  I find this the easiest way to modify a route policy for external BGP sessions and I see it used most for multi homed EBGP sessions.  This is widely used for route modifications in multihomed BGP setups.

Origin Code:
The origin code is a way to pick the best path based upon a known route having been learned as an ibgp, ebgp, or incomplete route.  The modification of this is limited so it's not popular to use in a the real world for complex route manipulation.  This is not heavily depended on in production use.

MED or Metric:
The MED (or Metric) value allows for route modification by appending a value associated to the route, similar to how OSPF or EIGRP might append a metric.  It is very similar to the local preference, but not worth configuring if local preference can be depended on.

Paths:
The Path attribute selects a route that was learned via an ebgp neighbor over an ibgp neighbor.  This is standard for most BGP setups and isn't used for multihomed ebgp setups.  This will not do much for true route manipulation.

Router ID:
This attribute just says that if all the criteria above is equal amongst 2 paths, it will select the path with the peer that has the lowest router ID, or IP address.  This is a last resort route manipulating method.

Ultimately the way that I have seen route manipulation done in live multihomed BGP environments is to either adjust (a) the local preference, (b) the AS path, or (c) the MED.   If you focus on those you will get the biggest bang for your buck with route manipulation.  

I'm going to put together a sample of a route filter based upon each method so that you can see how these work exactly.

In the meantime I hope this helps.  Please leave any questions below.  Thanks for reading!

David

Network Troubleshooting Techniques:

 

The 3 Main Approaches to Troubleshooting

As part of Cisco's CCNA certification they require you to know and understand 3 main methods of troubleshooting network issues from a client to a server.  The approaches are: the Top to Bottom Approach, the Cut-Through Approach, and the Bottom to Top Approach.  I will discuss the 3 of them as much as possible.  

For this you should have some knowledge of the OSI model.  There are 7 layers.  I remember them with this mnemonic: "Please Do Not Take Sales People Advice"  - Physical, Datalink, Network, Transport, Session, Presentation, and Application.  (That actually lists them from the bottom up.).  Below I briefly listed problems that can occur at each layer.  The main 3 layers you will look at when troubleshooting are the Application, Network, and Physical layers.  The rest of the layers will not have problems quite as often.

Quick Review of the OSI Model and problems that can occur at each layer.

1. Application - Many times applications themselves can be misconfigured.  The most important reference within the application to look at is the target IP and Port.  Once you have confirmed that these are within the configuration you should be good to go.
2. Presentation - You will rarely look for problems here.  This is how the applications code handles the network connection.  You can skip this when performing initial troubleshooting.
3. Session - This also does not need much attention.  This is how the operating system handles the network connection.  I recommend skipping this as well during the initial troubleshooting.
4. Transport - This is where the TCP/IP protocol begins to take shape.  Here is where you need to confirm that the right application port is being used.  If you find a problem in this layer it may mean that there is an issue with the TCP/IP installation on the system itself.  You may need to reinstall the TCP/IP protocol.  I have only seen a problem here twice in the last 10 years.  It is rare and you should troubleshoot it after going through everything else. 
5. Network - This is where you  configure your IP address, subnet mask, default gateway, and any static routes.  Many times there are problems with routes and IP configurations in the layer.  So you will want to inspect them closely on the system you are troubleshooting.
6. Datalink - This is where the NIC's MAC address is referenced.  It's also used for the ARP table.  When there is a problem here you will usually find that it is related to the MAC address that the system has assigned for another node on the network.  Every so often you will find a problem here, but not that often.     
7. Physical - This is all about the cabling.  Many times cables can be broken, ever so slightly, that you need to replace them entirely.  Transceivers also fall into this layer.  You will find that on occasion they do malfunction.

Top to Bottom Approach:  The Application Layer

For this approach we take the OSI model starting from application layer and attempt to troubleshoot the problem from there.  I've listed out basic steps for what to do.

1. Look at your application and try to recreate the problem that is happening.
2. Check your applications configuration and make sure that the target IP and Port are correctly configured.
3. Confirm if there are any proxies configured and if you application needs to use one.
4. Perform a "telnet test" (instructions below) from the client to the server.  Confirm if it is successful or not. *This will only work if you are troubleshooting a TCP application.
5. If your tests are unsuccessful, and the application configurations are correct, you will go down the OSI model to continue troubleshooting. 

How to do a telnet test:

(1st) Make sure that telnet is installed on the machine that you are running the test from

(2nd) Open up a command line on the client machine

(3rd) type "telnet <target server IP> <target application port>"

(*) Great way to test this is to do a telnet test to a website.  Try telneting to www.google.com on port 80 (the web service port).  The command will look like: "telnet www.google.com 80".  After you hit enter you should immediately see a blank screen or on a unix system it will tell you "Connected to www.google.com" and allow you to enter text.  

Cut Through Approach: The Network Layer

This method involves starting from the network layer of the OSI model.  I've listed the basic steps for what to do.

1. Try to ping the server's IP address.  
1. From a command line type: "ping <server IP>"
2. If ping is unsuccessful, then try a traceroute to the server.  The key here is to see what the last hop is along the path.  
1. From a Windows command line type: "tracert <server IP>"
2. From a Linux/UNIX command line type: "traceroute <server IP>
3. If you weren't able to ping the server and your traceroute shows the last hop as an IP address along the path, you need to investigate what the routing table on that last hop look like.  It may not be configured with the right next hop.  

  (*)  Something to keep in mind is that if there is a firewall along the path, the firewall may block the traceroute packet from everything after it.  This may confuse you when troubleshooting into thinking that the firewall is the problem.

Bottom to Top Approach: The Physical Layer

This method essentially involves starting from the Physical layer and working your way up when troubleshooting.  There is very little to starting here, and it is probably one of the easier ways to troubleshoot, at least initially.  

1. Check the cable at the client.
1. Make sure the cable is connected properly or even reseat the cable if necessary.
2. You may need to completely change out a cable.  Sometimes it is actually pretty difficult to tell if there is a problem with the cabling because it can be something so subtle that you don't even notice it.
2. Check the cable connection at the switch.
1. It's always good to check the cables connection to the switch to rule out any connection problem at that end.  You may want to reseat the cable here as well.
2. If your switch uses a transceiver for terminating the cable you may want to change out the entire transceiver.
3. You can always move the connection of the client to another port if you suspect there actually is a problem with a port.  This is not completely uncommon.
3. Check the cabling at your patch panel.
1. Many times connections may go through a patch panel and we need to make sure that the connections are seated properly
2. Similar to moving a connection on the switch you may want to move a connection across patch panel ports as part of troubleshooting.

*What Experience Teaches:

1. Make sure that only one person is having the problem that is being reported.  Confirm that no one else is experiencing the problem.  That can easily tell you if you need to troubleshoot from the client or the server.
2. Let users attempt the most basic troubleshooting techniques before you get involved.  (ie. have them ping the server, reseat the cable to their PC, reboot their PC)

Questions and comments are always appreciated.  I really hope this helps.  Thanks for reading.

- David Pagan

25 Things Every High Availability NOC Should Have

Below I've listed the top 25 things I believe every Network Operations Center (NOC) should have in order to become high availability.  This applies to NOCs that are operational 24/5 or require an uptime of 365 days a year.  I've broken this up into several sections to help you consider all aspects of what makes the NOC. 

Basics:

1. Hunt Groups and Call Forwarding
You should be able to have multiple phones all ring at once when calls come in.  When no one is in the NOC (very rare) it should forward to an on-call cell number.

2. Displays
Multiple displays can help to increase your NOCs overall monitoring capabilities.  Typically 4 monitors per person and at least a couple of large screen TV's can help to handle all of the areas of monitoring.  The large displays also help to demonstrate your monitoring capabilities to your upper management when they tour the NOC.

3. Automated Monitoring and Alerting 
Automated monitoring and alerting tools are probably the most crucial thing any NOC has to have.  You honestly don't even have a NOC without the right monitoring visibility into your environment.  There are countless free and paid for tools that will do alerting for you.  The best I can recommend are Intermapper and Orion NPM.  Intermapper will give you the best dashboard view and Orion can give you the most additional features.


Advancements:

4. Automated Reporting
Reports are one of my favorite ways to get statistics of what's really going on in an environment.  The thing about reports is that they mean nothing if there is no one to read them. This is where the NOC can really help to make the best of reporting.  You only need to run reports 2-3 times a day to have a great holistic view of your environment.  Reports should include not just numerical information, but also graphical analytics.

5. Historical Analytics
An important part of maintaining and supporting any network or system is having insight to the trending of statistics unique to what you're monitoring.  In order to do this you will need some kind of system that records trends over a period of time.  The typical retention on data should be at least 3 months if not longer.

6. Configuration Management
For network equipment and application configurations, it's crucial to be able to record and easily restore configuration files.  The best way to do this is to have a configuration management system.  This will manage the storage of configurations over the course of months and allow for easy downloading of configuration files or even running diffs against older configuration files and newer ones.


Communication Tactics for your NOC:

7. Paging System
Many people think that with the dawn of email on the phone, that there is no more need to a paging system.  However the best way to bypass email filters during an emergency, and get someone to pick up their phone is to send a page to them.  Most paging systems support groups similar to distribution lists.

8. Email Distribution Lists
You want to make sure that your NOC has it's own email list with the direct manager included.  Besides that all other levels of escalation should have their own email list.  The email list for the NOC should receive reports, alerts, and details communication strings with customers and vendors.  It's the single best way to make sure that everyone in your NOC is on the same page.

9. Company-wide Audible Alert System
This is an alert system that is manually triggered.   The way I've seen this implemented is using loud speakers to sound off a generally monotone alarm, different from a fire alarm but distinctive in its own way.  It allows the NOC to give notice to management and tier 3 engineers to report to the NOC for escalated troubleshooting.  It is probably the most effective way, along with paging, to get required attention.


Documentation:

10. Run Book
A run book is a living document that contains everything from most basic procedures to troubleshooting steps.  A hard copy of the run book are rare now or days,  they are best preserved in an online wiki.

11. Escalation Diagram
Your NOC needs to know exactly who to escalate any issues with and which teams to involve depending on the issue at hand.  Keep an escalation diagram available for them to reference.  These should be printed out and visually available within the work environment.

12. Escalation Contact List
Along with an Escalation Diagram you should have an updated contact list that your NOC can reference for emergencies.  Here you would contain cell and home numbers of the escalation contacts.

13. Service Provider and Vendor Contact List
The NOC should always have a copy of all of the vendor and service provider contacts.  This will include your sales rep in the event that the regular support lines aren't getting you anywhere.

14. Service Contracts
Your NOC needs to have access to all service contract information, serial numbers, and circuit IDs.  Without this information, in an updated manner, your NOC will have a tough time calling service providers and vendors to address urgent issues.

15. Full Hardware Inventory
One of the major initiatives that a NOC should partake in is the gathering of information for all the hardware within the environment.  They should have serial numbers, model numbers, and details specs of anything installed on the hardware as well.

16.  Datacenter Elevation Diagram
Many times when a NOC utilizes remote hands at the datacenter it helps to have an actual elevation diagram for everything that is located in the datacenter.  The elevation diagram will include the cage number, cabinet numbers, front and rear views, hardware vendor, model, and serial numbers.  These can be as simple as a spread sheet or as detailed as a visio diagram with vendor stencils.

Security Best Practices for the NOC:

17. Access cards for NOC access
Many companies, depending on the services they perform, require that their operations center be siloed off from the rest of the company.  While this may seem like an annoyance to many, it is one of the most basic ways to protect your production systems from intruders.  But in order for this to work it's best, the next point MUST be followed.

18. Separate Production and Corporate Networks
The best way I've ever seem this handled in a NOC was to give your operations personal 2 computers.  One was dedicated for corporate office access (i.e. email and IMs) and the other was dedicated to production access.  The production PC had no access to the internet or even internal email systems.  This insures that people who plug into an open port by a desk at some near by cubicle cannot reach your production systems.

19. Access to the internet from the NOC needs to be limited
Without the right tools, it can be extremely difficult to understand how much garbage traffic goes over the internet from our computers.  Unfortunately, not all of this traffic is just harmless spam and spyware.  Many times host machines are hijacked without users even knowing it.  These hijackings can lead to access of a seemingly secure network.  This is why this goes hand in hand with isolating your production and corporate networks.  It's better for a hacker to take down your email server than for them to take down your trading server.


Policies and Procedures:

20. Policies
As vague as this may sound, policies ranging from security standards, standard operating procedures in the NOC, change management policy, and an updated HR policy manual can clearly identify to your operations personal exactly how to conduct themselves within the NOC.  Remember that your policies are only as good as you enforce them to be.

21. Shift Turnovers
Every operational shift should provide a turnover to the next shift for a full understanding of all open and even closed items that the NOC has been working on.  These items should be held over for at least 24 hours so all shifts are aware of work that has happened throughout the entire day.

22. Ticketing System
A ticketing system will give you the ability to track old problems and incidents over the course of months or even years.  The best ticketing systems will even develop charts and statistics based on the tickets you've created and gives you a great way to track the productivity of your team.


Staffing and Support:

23. Diversifying your team
The best NOCs always have cross-training of some kind with regards to understanding and working with different systems and apps.  Among all of these things you will find that having your team know and understand these items will give you the best of all worlds: routing/switching, firewalls, Linux, Windows, server hardware, storage, virtualization, scripting, and application specific support.  These items cover just about everything your NOC will need to know.  You won't need expert level understanding at your NOC.  You'll only need people who have a low to mid level of understanding of the environment and their areas of specialty.  You should always be able to rely on your tier 3 engineers for high level support.

24. The right head count
The best way to calculate the number of heads needed is to look at a combination of the number of servers, network devices, employees, hours of usage for your production systems, and the quantity of customers.  I'm still working on a formula for this, so as soon as I get it together I will post it here.

25.  Remote Hands or Onsite Staff for Datacenter Support
When you need fast access to hardware at the datacenter, don't skim out on the datacenter support.  If you can't afford to have an actually datacenter technician on site during production hours make sure that your equipment is housed in a datacenter that provides "Hot Hands" or "Remote Hands".  This is definitely a life saver for the HA NOC.

Thanks for reading and I hope this helps you put together your NOC.  I'm definitely interested in feedback.

-David Pagan

How to configure a basic S2S VPN in Palo Alto

Here's a quick how to for setting up a very simple S2S IPSEC VPN on a Palo Alto.

1. Create a tunnel interface
• Here you will also create the security zone that will be used
2. Create the IKE Gateway and specify your tunnel’s security zone
• Include the Peer address and your outside interface
3. Create your IKE crypto policy
4. Create your IPSEC crypto policy
5. Create a new IPSEC Tunnel using the IKE/IPSEC Crypto policies you made.  
• Also create under this any Proxy IDs you may require to conform with a Cisco IPSEC VPN.  They're very similar to policies that you might create on a netscreen.
6. Create 2 new policies for the inbound and outbound traffic via the S2S VPN.
• These are typically the local and remote destination networks.  You need to put them in twice for inbound and outbound permissions.
7. Add the tunnel interface to the default virtual router and add the static routes to your remote network via your tunnel interface.

Remember that when creating a S2S with any Cisco device, you will need to use Proxy IDs.  They mimic the ACLs that Cisco uses to define interesting traffic and are needed during the initial P2 setup.

Enjoy!  Let me know if there are any question.  

- David Pagán

Port Redirection on Cisco ASA 5500 (8.4 Code)

I figure I would make this one of my first posts since this is a recent change I made at my network.  

How to do Port Redirection on a Cisco ASA running code train 8.4.x

First you need to identify your internal host and create an object to it:

object network obj-10.10.10.170
 host 10.10.10.170
!

Next you need to create the object for your outside natted address:

object network obj-20.20.20.170
 host 20.20.20.170
!


Now you create the actual internal port that you are redirecting.

object service obj-tcp-8080
 service tcp source eq 8080
!

Next we'll create the port mapping to the external natted address.

object service obj-tcp-80
 service tcp source eq 80
!

Once this is all complete we mesh everything all together and have a wonderful config like this:

nat (inside,outside) source static obj-10.10.10.170 obj-20.20.20.170 service obj-tcp-8080 obj-tcp-80


With this complete we have successfully created a port mapping.  As with any other static nat, make sure it is placed before the dynamic nat for all other traffic.  As so:

nat (inside,outside) source static obj-10.10.10.170 obj-20.20.20.170 service obj-tcp-8080 obj-tcp-80
nat (inside,outside) source dynamic any interface description DEFAULT_NAT

You can confirm that the configuration was successful by checking your xlate.

asa5510# show xlate
TCP PAT from inside:10.10.10.170 8080-8080 to outside:20.20.20.170 80-80
    flags sr idle 0:22:33 timeout 0:00:00

Hope this is helpful to others.  I got a lot of useful information from this link: https://supportforums.cisco.com/thread/2169361.

Thanks

David P.